{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Evaluation\n",
    "\n",
    "This notebook demonstrates how to use Cloud AI Platform to execute continuous evaluation of a deployed machine learning model. You'll need to have a project set up with Google Cloud Platform. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up\n",
    "Start by creating environment variables for your Google Cloud project and bucket. Also, import the libraries we'll need for this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change these to try this notebook out\n",
    "PROJECT = 'munn-sandbox'\n",
    "BUCKET = 'munn-sandbox'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['TFVERSION'] = '2.1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2.0\n"
     ]
    }
   ],
   "source": [
    "import shutil\n",
    "\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "\n",
    "from google.cloud import bigquery\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from tensorflow_hub import KerasLayer\n",
    "from tensorflow.keras.layers import Dense, Input, Lambda\n",
    "from tensorflow.keras.models import Model\n",
    "print(tf.__version__)\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and deploy the model\n",
    "For this notebook, we'll build a text classification model using the Hacker News dataset. Each training example consists of an article title and the article source. The model will be trained to classify a given article title as belonging to either `nytimes`, `github` or `techcrunch`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>led inventor and employer settle for $8.1m  2005</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yc-backed photo-sharing service picplum loses ...</td>\n",
       "      <td>techcrunch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>as facebook turns 10  zuckerberg wants to chan...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6 big healthtech ideas that will change medici...</td>\n",
       "      <td>techcrunch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>building the prince of persia sources to creat...</td>\n",
       "      <td>github</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title      source\n",
       "0  led inventor and employer settle for $8.1m  2005      nytimes\n",
       "1  yc-backed photo-sharing service picplum loses ...  techcrunch\n",
       "2  as facebook turns 10  zuckerberg wants to chan...     nytimes\n",
       "3  6 big healthtech ideas that will change medici...  techcrunch\n",
       "4  building the prince of persia sources to creat...      github"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DATASET_NAME = \"titles_full.csv\"\n",
    "COLUMNS = ['title', 'source']\n",
    "\n",
    "titles_df = pd.read_csv(DATASET_NAME, header=None, names=COLUMNS)\n",
    "titles_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We one-hot encode the label..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLASSES = {\n",
    "    'github': 0,\n",
    "    'nytimes': 1,\n",
    "    'techcrunch': 2\n",
    "}\n",
    "N_CLASSES = len(CLASSES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_labels(sources):\n",
    "    classes = [CLASSES[source] for source in sources]\n",
    "    one_hots = to_categorical(classes, num_classes=N_CLASSES)\n",
    "    return one_hots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 1., 0.],\n",
       "       [0., 0., 1.],\n",
       "       [0., 1., 0.],\n",
       "       [0., 0., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encode_labels(titles_df.source[:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and create a train/test split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "N_TRAIN = int(len(titles_df) * 0.80)\n",
    "\n",
    "titles_train, sources_train = (\n",
    "    titles_df.title[:N_TRAIN], titles_df.source[:N_TRAIN])\n",
    "\n",
    "titles_valid, sources_valid = (\n",
    "    titles_df.title[N_TRAIN:], titles_df.source[N_TRAIN:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, Y_train = titles_train.values, encode_labels(sources_train)\n",
    "X_valid, Y_valid = titles_valid.values, encode_labels(sources_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['led inventor and employer settle for $8.1m  2005 ',\n",
       "       'yc-backed photo-sharing service picplum loses one co-founder',\n",
       "       'as facebook turns 10  zuckerberg wants to change how tech industry works'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Swivel Model\n",
    "\n",
    "We'll build a simple text classification model using a Tensorflow Hub embedding module derived from Swivel. [Swivel](https://arxiv.org/abs/1602.02215) is an algorithm that essentially factorizes word co-occurrence matrices to create the words embeddings. \n",
    "TF-Hub hosts the pretrained [gnews-swivel-20dim-with-oov](https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim-with-oov/1) 20-dimensional Swivel module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "SWIVEL = \"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim-with-oov/1\"\n",
    "swivel_module = KerasLayer(SWIVEL, output_shape=[20], input_shape=[], dtype=tf.string, trainable=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `build_model` function is written so that the TF Hub module can easily be exchanged with another module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(hub_module, model_name):\n",
    "    inputs = Input(shape=[], dtype=tf.string, name=\"text\")\n",
    "    module = hub_module(inputs)\n",
    "    h1 = Dense(16, activation='relu', name=\"h1\")(module)\n",
    "    outputs = Dense(N_CLASSES, activation='softmax', name='outputs')(h1)\n",
    "    model = Model(inputs=inputs, outputs=[outputs], name=model_name)\n",
    "    \n",
    "    model.compile(\n",
    "        optimizer='adam',\n",
    "        loss='categorical_crossentropy',\n",
    "        metrics=['accuracy']\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_and_evaluate(train_data, val_data, model, batch_size=5000):\n",
    "    tf.random.set_seed(33)    \n",
    "    X_train, Y_train = train_data\n",
    "\n",
    "    history = model.fit(\n",
    "        X_train, Y_train,\n",
    "        epochs=100,\n",
    "        batch_size=batch_size,\n",
    "        validation_data=val_data,\n",
    "        callbacks=[EarlyStopping()],\n",
    "    )\n",
    "    return history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "txtcls_model = build_model(swivel_module, model_name='txtcls_swivel')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"txtcls_swivel\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "text (InputLayer)            [(None,)]                 0         \n",
      "_________________________________________________________________\n",
      "keras_layer (KerasLayer)     (None, 20)                389380    \n",
      "_________________________________________________________________\n",
      "h1 (Dense)                   (None, 16)                336       \n",
      "_________________________________________________________________\n",
      "outputs (Dense)              (None, 3)                 51        \n",
      "=================================================================\n",
      "Total params: 389,767\n",
      "Trainable params: 389,767\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "txtcls_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and evaluation the model\n",
    "With the model defined and data set up, next we'll train and evaluate the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up train and validation data\n",
    "train_data = (X_train, Y_train)\n",
    "val_data = (X_valid, Y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For training we'll call `train_and_evaluate` on `txtcls_model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "16/16 [==============================] - 1s 45ms/step - loss: 1.3334 - accuracy: 0.3129 - val_loss: 1.2283 - val_accuracy: 0.3434\n",
      "Epoch 2/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 1.1785 - accuracy: 0.3797 - val_loss: 1.1155 - val_accuracy: 0.4178\n",
      "Epoch 3/100\n",
      "16/16 [==============================] - 1s 34ms/step - loss: 1.0804 - accuracy: 0.4463 - val_loss: 1.0397 - val_accuracy: 0.4774\n",
      "Epoch 4/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 1.0114 - accuracy: 0.4988 - val_loss: 0.9824 - val_accuracy: 0.5200\n",
      "Epoch 5/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.9571 - accuracy: 0.5420 - val_loss: 0.9346 - val_accuracy: 0.5554\n",
      "Epoch 6/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.9109 - accuracy: 0.5765 - val_loss: 0.8926 - val_accuracy: 0.5882\n",
      "Epoch 7/100\n",
      "16/16 [==============================] - 0s 31ms/step - loss: 0.8691 - accuracy: 0.6067 - val_loss: 0.8539 - val_accuracy: 0.6149\n",
      "Epoch 8/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.8297 - accuracy: 0.6313 - val_loss: 0.8169 - val_accuracy: 0.6358\n",
      "Epoch 9/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.7917 - accuracy: 0.6526 - val_loss: 0.7813 - val_accuracy: 0.6539\n",
      "Epoch 10/100\n",
      "16/16 [==============================] - 1s 34ms/step - loss: 0.7549 - accuracy: 0.6729 - val_loss: 0.7475 - val_accuracy: 0.6711\n",
      "Epoch 11/100\n",
      "16/16 [==============================] - 1s 31ms/step - loss: 0.7198 - accuracy: 0.6902 - val_loss: 0.7160 - val_accuracy: 0.6870\n",
      "Epoch 12/100\n",
      "16/16 [==============================] - 0s 31ms/step - loss: 0.6869 - accuracy: 0.7059 - val_loss: 0.6869 - val_accuracy: 0.7031\n",
      "Epoch 13/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.6566 - accuracy: 0.7204 - val_loss: 0.6608 - val_accuracy: 0.7160\n",
      "Epoch 14/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.6292 - accuracy: 0.7327 - val_loss: 0.6376 - val_accuracy: 0.7266\n",
      "Epoch 15/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.6048 - accuracy: 0.7434 - val_loss: 0.6173 - val_accuracy: 0.7342\n",
      "Epoch 16/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.5829 - accuracy: 0.7542 - val_loss: 0.5996 - val_accuracy: 0.7412\n",
      "Epoch 17/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.5635 - accuracy: 0.7634 - val_loss: 0.5843 - val_accuracy: 0.7486\n",
      "Epoch 18/100\n",
      "16/16 [==============================] - 1s 32ms/step - loss: 0.5461 - accuracy: 0.7716 - val_loss: 0.5711 - val_accuracy: 0.7543\n",
      "Epoch 19/100\n",
      "16/16 [==============================] - 0s 31ms/step - loss: 0.5306 - accuracy: 0.7788 - val_loss: 0.5593 - val_accuracy: 0.7585\n",
      "Epoch 20/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.5166 - accuracy: 0.7854 - val_loss: 0.5492 - val_accuracy: 0.7636\n",
      "Epoch 21/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.5037 - accuracy: 0.7914 - val_loss: 0.5403 - val_accuracy: 0.7692\n",
      "Epoch 22/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.4921 - accuracy: 0.7969 - val_loss: 0.5321 - val_accuracy: 0.7738\n",
      "Epoch 23/100\n",
      "16/16 [==============================] - 0s 31ms/step - loss: 0.4812 - accuracy: 0.8023 - val_loss: 0.5250 - val_accuracy: 0.7768\n",
      "Epoch 24/100\n",
      "16/16 [==============================] - 0s 31ms/step - loss: 0.4711 - accuracy: 0.8067 - val_loss: 0.5186 - val_accuracy: 0.7793\n",
      "Epoch 25/100\n",
      "16/16 [==============================] - 1s 33ms/step - loss: 0.4617 - accuracy: 0.8109 - val_loss: 0.5129 - val_accuracy: 0.7814\n",
      "Epoch 26/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.4529 - accuracy: 0.8148 - val_loss: 0.5077 - val_accuracy: 0.7849\n",
      "Epoch 27/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.4446 - accuracy: 0.8184 - val_loss: 0.5032 - val_accuracy: 0.7882\n",
      "Epoch 28/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.4367 - accuracy: 0.8217 - val_loss: 0.4990 - val_accuracy: 0.7910\n",
      "Epoch 29/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.4294 - accuracy: 0.8253 - val_loss: 0.4954 - val_accuracy: 0.7934\n",
      "Epoch 30/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.4224 - accuracy: 0.8285 - val_loss: 0.4923 - val_accuracy: 0.7953\n",
      "Epoch 31/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.4158 - accuracy: 0.8321 - val_loss: 0.4893 - val_accuracy: 0.7964\n",
      "Epoch 32/100\n",
      "16/16 [==============================] - 1s 37ms/step - loss: 0.4095 - accuracy: 0.8348 - val_loss: 0.4868 - val_accuracy: 0.7978\n",
      "Epoch 33/100\n",
      "16/16 [==============================] - 1s 33ms/step - loss: 0.4036 - accuracy: 0.8377 - val_loss: 0.4847 - val_accuracy: 0.7990\n",
      "Epoch 34/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.3980 - accuracy: 0.8401 - val_loss: 0.4827 - val_accuracy: 0.8004\n",
      "Epoch 35/100\n",
      "16/16 [==============================] - 0s 30ms/step - loss: 0.3927 - accuracy: 0.8420 - val_loss: 0.4814 - val_accuracy: 0.8007\n",
      "Epoch 36/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.3876 - accuracy: 0.8447 - val_loss: 0.4800 - val_accuracy: 0.8020\n",
      "Epoch 37/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.3828 - accuracy: 0.8471 - val_loss: 0.4791 - val_accuracy: 0.8022\n",
      "Epoch 38/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.3782 - accuracy: 0.8487 - val_loss: 0.4780 - val_accuracy: 0.8038\n",
      "Epoch 39/100\n",
      "16/16 [==============================] - 1s 32ms/step - loss: 0.3738 - accuracy: 0.8505 - val_loss: 0.4775 - val_accuracy: 0.8029\n",
      "Epoch 40/100\n",
      "16/16 [==============================] - 1s 32ms/step - loss: 0.3697 - accuracy: 0.8521 - val_loss: 0.4769 - val_accuracy: 0.8033\n",
      "Epoch 41/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.3657 - accuracy: 0.8540 - val_loss: 0.4767 - val_accuracy: 0.8041\n",
      "Epoch 42/100\n",
      "16/16 [==============================] - 0s 29ms/step - loss: 0.3619 - accuracy: 0.8561 - val_loss: 0.4768 - val_accuracy: 0.8047\n"
     ]
    }
   ],
   "source": [
    "txtcls_history = train_and_evaluate(train_data, val_data, txtcls_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f7af06f7c10>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1f3/8deZbJN9IXtCFvYAAZSAAgoiKLhBXUEQlNpaRXFppbYu39qqP63aWhcsVUsRxQIFKrQuqGCNyhowyB7DFrLvG9lnzu+PO0KAAAGS3Mzk83w87uPOXWbmM/ehb27OPfdcpbVGCCGE87OYXYAQQoi2IYEuhBAuQgJdCCFchAS6EEK4CAl0IYRwEe5mfXFoaKhOSEgw6+uFEMIpbd26tVhrHdbSNtMCPSEhgbS0NLO+XgghnJJS6vDptkmTixBCuAgJdCGEcBES6EII4SJMa0MXQnRNjY2NZGdnU1dXZ3YpnZrVaiU2NhYPD49Wv0cCXQjRobKzs/H39ychIQGllNnldEpaa0pKSsjOziYxMbHV75MmFyFEh6qrq6Nbt24S5meglKJbt27n/FeMBLoQosNJmJ/d+Rwjpwv0fflVPPfRbmobbGaXIoQQnYrTBXp2WQ1vf32Q9CPlZpcihHBSfn5+ZpfQLpwu0FPiQwBIO1RqciVCCNG5OF2gB/p40DfCny2Hy8wuRQjh5LTWzJ07l4EDB5KcnMzSpUsByMvLY/To0QwZMoSBAwfy9ddfY7PZuOuuu47t+8orr5hc/amcsttiSkIwq9Jzsdk1bha5uCKEs/r9f3axO7eyTT+zf3QAv7thQKv2XblyJenp6Wzfvp3i4mKGDRvG6NGj+eCDD5gwYQJPPPEENpuNmpoa0tPTycnJYefOnQCUl3e+Zl+nO0MHGJ4YQnV9E3vy2vY/BCFE1/LNN99w++234+bmRkREBGPGjGHLli0MGzaMf/zjHzz99NPs2LEDf39/evTowYEDB5gzZw6ffvopAQEBZpd/Cic9Qz/ejj4wJtDkaoQQ56u1Z9LtRWvd4vrRo0eTmprKRx99xIwZM5g7dy4zZ85k+/btrFmzhnnz5rFs2TIWLFjQwRWfmVOeoccEeRMdaJV2dCHEBRk9ejRLly7FZrNRVFREamoqw4cP5/Dhw4SHh/Pzn/+cu+++m23btlFcXIzdbufmm2/mmWeeYdu2bWaXfwqnPEMHGJYYwob9JWit5SYFIcR5ufHGG9mwYQODBw9GKcWLL75IZGQk7777Li+99BIeHh74+fmxaNEicnJymDVrFna7HYDnn3/e5OpPpU73J0d7S0lJ0RfygIv3Nh7mqQ93kjp3LHHdfNqwMiFEe9qzZw9JSUlml+EUWjpWSqmtWuuUlvZ3yiYXgGEJwQBskf7oQggBOHGg9wn3J8DqLoEuhBAOThvoFosiJSFEAl0IIRycNtDBuMFof9FRSqrrzS5FCCFM59SBPvzH/ujSfVEIIZw70JNjA/F0t8hAXUIIgZMHupe7G4NjA9lySM7QhRDCqQMdYFhCCDtzKqhpaDK7FCGECzrT2OmHDh1i4MCBHVjNmblEoDfZtTzwQgjR5Tntrf8/ujg+GKUg7VAZI3uGml2OEOJcfPIbyN/Rtp8ZmQzXvHDazY899hjx8fHMnj0bgKeffhqlFKmpqZSVldHY2Mizzz7L5MmTz+lr6+rquO+++0hLS8Pd3Z0///nPjB07ll27djFr1iwaGhqw2+2sWLGC6OhobrvtNrKzs7HZbDz11FNMmTLlgn42uECgB3o7HnghF0aFEK0wdepUHn744WOBvmzZMj799FMeeeQRAgICKC4u5tJLL2XSpEnnNE7UvHnzANixYwd79+7l6quvJiMjg/nz5/PQQw8xffp0GhoasNlsfPzxx0RHR/PRRx8BUFFR0Sa/zfkCXWso3APhSeA42MMSQli5LZsmmx13N6dvRRKi6zjDmXR7ueiiiygsLCQ3N5eioiKCg4OJiorikUceITU1FYvFQk5ODgUFBURGRrb6c7/55hvmzJkDQL9+/YiPjycjI4MRI0bw3HPPkZ2dzU033UTv3r1JTk7m0Ucf5bHHHuP666/n8ssvb5Pf5nzpl74Y/joCivYdW5WSEMzRBht786tMLEwI4SxuueUWli9fztKlS5k6dSqLFy+mqKiIrVu3kp6eTkREBHV1def0macb6HDatGmsXr0ab29vJkyYwLp16+jTpw9bt24lOTmZ3/72t/zhD39oi5/lhIHec5wx37P62KphjhuMpNlFCNEaU6dOZcmSJSxfvpxbbrmFiooKwsPD8fDw4Msvv+Tw4cPn/JmjR49m8eLFAGRkZJCVlUXfvn05cOAAPXr04MEHH2TSpEl8//335Obm4uPjwx133MGjjz7aZmOrO1+gB0RB90tg9/FAjw7yJibIWwJdCNEqAwYMoKqqipiYGKKiopg+fTppaWmkpKSwePFi+vXrd86fOXv2bGw2G8nJyUyZMoWFCxfi5eXF0qVLGThwIEOGDGHv3r3MnDmTHTt2MHz4cIYMGcJzzz3Hk08+2Sa/66zjoSulFgDXA4Va61M6XCqlpgOPORargfu01tvP9sUXNB76+jfgsyfgwe8gpAcADy/5jm/3l7D58XHywAshOjEZD7312mM89IXAxDNsPwiM0VoPAp4B3mpdqRcg6QZjvuc/x1alJIRQVFVPVmlNu3+9EEJ0Rmft5aK1TlVKJZxh+/pmixuB2Asv6yyC4yFqiNHsMuohAIYnGu3omw+WEt/Nt91LEEJ0HTt27GDGjBknrPPy8mLTpk0mVdSytu62eDfwyek2KqXuAe4BiIuLu7BvSroB1j0DFTkQGEOvMD8CvT1IO1TGrSndL+yzhRDtytmeBZycnEx6enqHfuf5PB60zS6KKqXGYgT6Y6fbR2v9ltY6RWudEhYWdmFf2N9xF5ej2cViUaTEB7PlsFwYFaIzs1qtlJSUnFdgdRVaa0pKSrBaref0vjY5Q1dKDQLeAa7RWpe0xWeeVWhvCEsyAv3SewEYlhjC2r2FFFfXE+rn1SFlCCHOTWxsLNnZ2RQVFZldSqdmtVqJjT23FuwLDnSlVBywEpihtc640M87J/0nQepLUF0EfmHHHhyddqiMiQNbf4eXEKLjeHh4kJiYaHYZLumsTS5KqX8CG4C+SqlspdTdSql7lVL3Onb5P6Ab8KZSKl0pdZ59Ec9D0iTQdtj7XwAGxgTi5W5h44GO+SNBCCE6k9b0crn9LNt/BvyszSo6FxEDIDjRuGs0ZRZe7m6M6RPGf7/P48nrkmRcFyFEl+LciaeU0exyMBVqjacW3Tw0luLqelJ/kPY5IUTX4tyBDpA0GexNsO9TAMb2DSfYx4MVW3NMLkwIITqW8wd6zMUQEHtssC5PdwuTh8Tw+e4CKmoaTS5OCCE6jvMHulLGTUaZa6HeGD73lqGxNNjs/Of7XJOLE0KIjuP8gQ5GoNvq4YfPARgQHUDfCH9WbMs2uTAhhOg4rhHocZeCb9ixZhelFDcPjeG7rHL2F1WbXJwQQnQM1wh0ixv0ux4yPoPGWgB+MiQGi4KVcpYuhOgiXCPQwei+2HgU9q8DIDzAyug+YazcloPNLmNGCCFcn+sEesLlYA06YYz0W4bGkldRx4b9cueoEML1uU6gu3lA32th38fQ1ADA+KQIAqzucnFUCNEluE6gg9HsUlcBh1IBsHq4cf3gaD7dmU91fZPJxQkhRPtyrUDvMRY8/WH70mOrbr44ltpGGx/vyDOxMCGEaH+uFegeVrh4BuxcAaUHAbg4LojEUF9WbJVmFyGEa3OtQAcYOcfoxvjtq4CjT/rFMWw6WMoReYC0EMKFuV6gB0TDkOmQvhgqjVv/b7w4FqWQi6NCCJfmeoEOcNnDYLfB+jcAiAnyZmTPbqzcliPPMRRCuCzXDPTgBEi+FdIWwNFiwLg4mlVaw5ZDZebWJoQQ7cQ1Ax3g8l9CUx1sfBOAiQMj8fV0k4ujQgiX5bqBHtbX6Je++W2oLcfH051rkqP4aEeejJMuhHBJrhvoAJf/CuorYcvbAPx0VCLV9U38LXW/yYUJIUTbc+1AjxoMva+GDW9CfTX9owOYPCSaf3x7iMKqOrOrE0KINuXagQ4wei7UlsLWhQD88qo+NNrsvLEu09y6hBCijbl+oHcfbozEuP51aKwjvpsvU4Z154NNWWSVyI1GQgjX4fqBDjD6UajON242Ah4c1xt3N8UrX2SYXJgQQrSdrhHoiWMgdhh88xewNRIRYOWukYl8mJ7D3vxKs6sTQog20TUCXSm4/FGoyIId/wLgvjE98fdy5+U1+0wuTggh2kbXCHSAPhMgIhm+/hPYGgn08eAXY3ryxZ5Cth4uNbs6IYS4YF0n0JWCsY9DSSZsMMZ4mTUqgVA/L/746T4Z40UI4fS6TqAD9LsW+l0P/3sBSvbj4+nOQ+N6sflgKV9lFJldnRBCXJCuFegA174Mbp7w34dBa6YMi6N7iDcvrdmH3S5n6UII59X1Aj0gCsY/DQdTIX0xnu4WfnlVH3blVvKRPKZOCOHEul6gAwydBXEjYM0TUF3IpMEx9Iv058+fZ9Bos5tdnRBCnJezBrpSaoFSqlAptfM025VS6jWlVKZS6nul1MVtX2Ybs1jghlehsQY+/Q1uFsWjV/flYPFRFm88bHZ1QghxXlpzhr4QmHiG7dcAvR3TPcBfL7ysDhDW1xiNcecKyPiMcUnhjO4Txotr9smQAEIIp3TWQNdapwJn6qg9GVikDRuBIKVUVFsV2K4uewRC+8JHv0Q1HOWFm5JxU4pHl2+XC6RCCKfTFm3oMcCRZsvZjnWnUErdo5RKU0qlFRV1gm6C7l4w6TWoOALrniU6yJunbujP5oOlvLvhkNnVCSHEOWmLQFctrGvx9FZr/ZbWOkVrnRIWFtYGX90G4i6FlLth03zI3sqtQ2MZ2zeMP366l4PFR82uTgghWq0tAj0b6N5sORbIbYPP7Tjjfwf+UbB6DsrexPM3DcLTzcLcf23HJk0vQggn0RaBvhqY6ejtcilQobV2rg7d1kC47mUo3AVfvUhkoJWnJw0g7XAZ//j2oNnVCSFEq7Sm2+I/gQ1AX6VUtlLqbqXUvUqpex27fAwcADKBt4HZ7VZte+p3HQyZDqkvQeYX3HhRDFf1j+ClNfvILKw2uzohhDgrZdagVCkpKTotLc2U7z6thhp4ZzxU5cG9X1NoCeXqV1JJ6ObL8ntH4O7WNe/DEkJ0HkqprVrrlJa2SUI15+kDty0CWyP8axbhPm78YfJA0o+U8/bX0vQihOjcJNBPFtoLJr8O2Zvh899xw6AorhkYySufZ5BRUGV2dUIIcVoS6C0ZcCMM/wVsnIfas5pnfjIQP6s7jyxNp67RZnZ1QgjRIgn007n6WYhJgVUPEFqfzYs3D2JXbiVPfbhTHoYhhOiUJNBPx90Tbl0IFjdYNpPxvQN48Mpe/GtrNh9szjK7OiGEOIUE+pkEdYeb3oaCnfDxXB4a34cxfcJ4evUutmWVmV2dEEKcQAL9bHpfBZc/Ct+9h9v2D3h16hAiA63c9/5Wiqrqza5OCCGOkUBvjbGPQ+Jo+O8jBBVv4293pFBR28j9H2yTB2IIIToNCfTWsLjBre8aTTD/nEp/ryKevymZzQdLef7jvWZXJ4QQgAR66/mEwPR/gbLA+zdzYx8rd41MYMG3B1mVnmN2dUIIIYF+TkJ6wO1LjaEB/nk7T1ydwPCEEB5b8T178irNrk4I0cVJoJ+r7sPgprcgewseq+7ljWmDCbB68Iv3tlJe02B2dUKILkwC/Xz0n2zceLRnNeEbnuOvd1xMfkUd97y3lfomuZNUCGEOCfTzNeJ+GH4PbHiDoQUreOnWQWw+WMrcf30vzyMVQphCAv18KQUTX4A+18Anv2ay9/fMndCX1dtzefmzfWZXJ4TogiTQL4TFDW75O0QOguU/ZXavMm4fHseb/9vP4k2Hza5OCNHFSKBfKE9fmLYM/MJRi2/hmUthbN8wnvpwJ1/uLTS7OiFEFyKB3hb8I2DmKvDwxn3xTcybGET/6ADu/2AbO3MqzK5OCNFFSKC3leAEmPEh2JvwWXIzC2+KJtjHk1kLt5BdVmN2dUKILkACvS2F94MZK6GunNCVU3hvSiJ1jTZm/WMLFbWNZlcnhHBxEuhtLfoimLYUKrLpsWYG70zpzaGSo/x8UZo87UgI0a4k0NtD/EiY8j4U7uWS9ffyl5v6sOVQKQ988B1NMjqjEKKdSKC3l97j4eZ3IHsL1+16lGev780Xewr49Qq58UgI0T7czS7ApQ34CTRUw6r7me75e8rGPcnLaw8Q5O3JU9cnoZQyu0IhhAuRQG9vF90BDTXwyVzuT7JQNvJR/v7tQYJ9PJgzrrfZ1QkhXIgEeke45B6wN6HW/JYn+7tRMeQB/vR5BkG+nsy4NN7s6oQQLkICvaOMmA3ahvrsSV4cYKGi3z3836qdBFjdmTwkxuzqhBAuQAK9I42cA/YmLF88zfyBbkyLv5NfLdtOgLcHY/uGm12dEMLJSS+XjnbZI3Dlk7jtXMZ7oe/RL8KX+97fyuaDpWZXJoRwchLoZhg9F654HM+dS1geu5TYQC9+unAL6UfKza5MCOHEJNDNcsVjMPrXWHcsZnXiSrr5uDPz75vYlSuDeQkhzo8EupnGPg6X/wqfHe/xUeJy/DwtzPj7ZjILq8yuTAjhhFoV6EqpiUqpfUqpTKXUb1rYHqiU+o9SartSapdSalbbl+qClIIrn4LRv8Zv9wesSVyCO3amvb2JQ8VHza5OCOFkzhroSik3YB5wDdAfuF0p1f+k3e4HdmutBwNXAH9SSnm2ca2uSSm48gkY+yT++5bzReJidFMD09/ZRE55rdnVCSGcSGvO0IcDmVrrA1rrBmAJMPmkfTTgr4x72f2AUqCpTSt1dWPmwvjfE5C5ii8SFlFTV8v0tzdSWFlndmVCCCfRmkCPAY40W852rGvuDSAJyAV2AA9prU8ZVlApdY9SKk0plVZUVHSeJbuwyx6GCc8TePATvuy+gLKqaqa/s4mS6nqzKxNCOIHWBHpLI0idPFzgBCAdiAaGAG8opQJOeZPWb2mtU7TWKWFhYedcbJcwYjZc+zJBR77gf7FvU1BazvR3NlF6tMHsyoQQnVxrAj0b6N5sORbjTLy5WcBKbcgEDgL92qbELmj4z+GG1wjOTeWr2PnkFpcy/Z1NlEmoCyHOoDWBvgXorZRKdFzonAqsPmmfLGAcgFIqAugLHGjLQrucoXfCT/5KcMFGvo56ncKiAqa/s4nyGgl1IUTLzhroWusm4AFgDbAHWKa13qWUulcpda9jt2eAkUqpHcBa4DGtdXF7Fd1lDLkdbllAYMl2vgp9ifLCbAl1IcRpKa3NeXpOSkqKTktLM+W7nU7mF7B0BjVeoVxf9ii+kT15/+5LCPTxMLsyIUQHU0pt1VqntLRN7hR1Br3Gw8xV+DRV8knAc9jydzNjwSYqahvNrkwI0YlIoDuL7sNh1id4uSlW+T6HZ/42Zv59E5V1EupCCIMEujOJ6A93r8HDN5gl1ucJzv+GGX/fTEWNhLoQQgLd+QQnwE/X4N6tBws8XyYu7zOmvbNR+qkLISTQnZJ/BNz1EZaYobzm/iqjipYxZf56GSZAiC5OAt1ZeQfBjH+jkq7ncbdF3FX5Jrf/7VtyZUAvIbosCXRn5ukDty6CkXOYrtbwVPWz3Dl/HVklNWZXJoQwgQS6s7NY4Opn4bo/McaSzmt1TzB7/kfsL6o2uzIhRAeTQHcVw36GmraMPh6FvNP4GI/PX8Le/EqzqxJCdCAJdFfS+yrcfvopob4eLLA9yat/m892efC0EF2GBLqriRqE+z3r8AhN5HX9PP95+2lS9xWaXZUQogNIoLuiwBg8f/4Zth7jedLyDwrf/xmr02TwSyFcnQS6q/Lyx+uOpdRd9mtucfuK+NU388Fn682uSgjRjiTQXZnFgnX8EzTetpg+7gVc/e0UFv3zfex2c0bYFEK0Lwn0LsCj//V43fs/tDWYaXvn8OH8p2hotJldlhCijUmgdxGW8D6EPvI1WWGjuanwdTa/chvV1VVmlyWEaEMS6F2IsgbSY/a/2dl3DiOPrqXwlcspOrDd7LKEEG1EAr2rsVgYePuz7LjiLYKaivFfNJ6sz14Hk55cJYRoOxLoXdTgsbdRfuf/+N7Sn7j1T5L3t5vgaInZZQkhLoAEehfWo0cvev1yDe8G/IKQvFSq/zIM+w/rzC5LCHGeJNC7uBA/K7c/+ALz+7xNbr0Vy+Ibafz4cWiqN7s0IcQ5kkAXeLpbeHDajWwcv5z3bFfhsXkeDX+7EvJ3mF2aEOIcSKALAJRSzBzdn7gZf+UBfk11URb6b2Pgi6ehUR6aIYQzkEAXJxjTJ4yHZz/E3X7zWNE0Cr55Bf3XkXDgK7NLE0KchQS6OEWvcD/en3Mt3wz4A7c3PEFhRR0smgQf3g81pWaXJ4Q4DQl00SJfL3demTKEyT+ZylX1L7DQciN6+z9h3nDYsVz6rQvRCUmgi9NSSjF1eBxL7x/LIp+7uK7+WQpUGKy4G967EfJ3ml2iEKIZCXRxVklRAayecxm9kkcwovhx3gu6D3vudzD/Mlh1P1TmmV2iEAIJdNFKfl7uvDp1CM/eOJhniscwvvEVDva+E7Yvhdcvhi+fh3p5MLUQZpJAF62mlGLaJXGsfmAUPkFhjN1xNb+PX0hDj/Hw1Qvw+lDYtgjsMjSvEGaQQBfnrF9kAP+ePYpfXtWH9zMsjNw/k41jl0BQHKyeYzTF7FwpwS5EB2tVoCulJiql9imlMpVSvznNPlcopdKVUruUUtJp2cV5uFl4cFxvVj9wGeH+VqZ+Yuch3z9SPekdsDXC8lnw5qVGk4ytyexyhegSlD5L9zOllBuQAVwFZANbgNu11rub7RMErAcmaq2zlFLhWuszPmo+JSVFp6WlXWj9ohNotNmZ92Umb6zLJMjHk//3kySuZiOkvgyFuyE4ES7/JQyaCu6eZpcrhFNTSm3VWqe0tK01Z+jDgUyt9QGtdQOwBJh80j7TgJVa6yyAs4W5cC0ebhYeHt+HVQ+MItTPk3veT+fn2+I5MuVzmPoBWAONppjXLoLNb8tQAkK0k9YEegxwpNlytmNdc32AYKXU/5RSW5VSM1v6IKXUPUqpNKVUWlFR0flVLDqtAdGBrH7gMh6b2I9vfihm/Ctf81pOH+pmrYXpKyAwBj5+FP6cBJ89BWWHzC5ZCJfSmkBXLaw7uZ3GHRgKXAdMAJ5SSvU55U1av6W1TtFap4SFhZ1zsaLz83S3cN8VPVn7qzGMT4rgz59nMOHVr1lnS4afroG7PobE0bBhHrw6BBbfBj98Dna72aUL4fRaE+jZQPdmy7FAbgv7fKq1Pqq1LgZSgcFtU6JwRtFB3sybfjHv330JbhbFTxem8bNFWzkScBHctgge3gGj50Lud7D4FqMv+/rXZawYIS5Aay6KumNcFB0H5GBcFJ2mtd7VbJ8k4A2Ms3NPYDMwVWt92nvD5aJo19HQZGfBtwd5be0P2Oyan12eyC/G9CTA6gFNDbBntdG2fmQjuHlB34nGBdRe4+UiqhAnOdNF0bMGuuMDrgX+ArgBC7TWzyml7gXQWs937DMXmAXYgXe01n8502dKoHc9eRW1PP/xXlZvzyXIx4PZV/Rk5ogErB5uxg75O2Dbe7BzBdQUg3cIDLwZBk+FmKGgWmr9E6JrueBAbw8S6F3XzpwKXlyzj9SMIqICrTw8vjc3XxyLu5ujBdDWCPvXwfYlsO9jaKqDbr1g0BQYcCOE9jb3BwhhIgl00Smt31/Mi5/uI/1IOT3DfJk7oS8TBkSimp+J11XA7tXw/VI49LWxLiwJkm6A/pMgYqCcuYsuRQJddFpaa9bsKuClNXvZX3SUQbGB3D+2F1clRWCxnBTUFTmw97+w5z9w+FvQdghOMMI9abLRLGOR0SyEa5NAF51ek83Oym05vPFlJlmlNfQO92P22J7cMCj6eFNMc9VFRnPMntXG4/HsjeAbDr3GQc9x0HMs+IZ2/A8Rop1JoAun0WSz89GOPN78cj/7CqqIDfbmF2N6cuvQ2OMXT09WWw4Za+CHNbD/S6gtBRREDzHCvdc4iB0Gbh4d+luEaA8S6MLp2O2atXsLmfdlJulHygn18+LuyxKZdkkcgd5nCGa7DXLTYf9ayFwL2VtA28DTH+IuhYRRED8Koi+SgBdOSQJdOC2tNRsOlPDml/v5JrMYH083brwohrtGJtA7wv/sH1BbDge/Ms7cD6+H4n3Geg8f46w94TKIH2m0v3t4t++PEaINSKALl7Azp4KF6w+xensuDU12RvXqxp0jEhiXFIHbyRdQT6e6CLLWw6FvjYAv2AlosHhAZDJ0H24EfffhENhdetCITkcCXbiUkup6lmw5wvsbD5NXUUdssDczLo1nyrDuBPmc452lNaWQtRGObDKaZ3K2QZNjNEi/SOg+zAj4qCEQNRi8g9r+BwlxDiTQhUtqstn5bHcBC9cfYvPBUjzdLUwYEMltKbGM6hl6arfH1rA1GmftR7ZA9mY4shnKDx/fHpxwPNyjBhuvfbu12W8S4mwk0IXL25NXyZLNWXyYnktFbSMxQd7cMjSWW4bG0j3E58I+/Ggx5G2HvHTHfPuJQ//6R0F4f4job8zD+0NYX2mTF+1CAl10GXWNNj7fXcCytCN8k1mM1jCqVzduS+nO1f0j8fY8TdfHc1VbZow9k5tuPJWpYBcU7QNbvbFdWSCkB4QnQWhfCO1jDFkQ2ge8/NqmBtElSaCLLimnvJYVW7NZlnaE7LJafDzdmDAgkklDormsVygeLd2wdCFsTVB6wAj4YyG/F0oPGl0nfxQQ4wj3vkbo/zgFxcnokuKsJNBFl2a3azYfKmVVeg4ffZ9HZV0TIb6eXJccxeQh0VwcF3x+7e2t1dQAZQehOMMx/WCczRf/AA1Vx/dTFgiMPR7wwQlGyAfFQVA8+HSTXjdCAl2IH9U32UjNKObD9By+2F1AfZOdmCBvrh8UxYSBkQyJDWrfcG9Oa6N9vuygcWbffCrZD/CAIssAAAyYSURBVHXlJ+7v7t0s4Lsb4R8Qa8wDY8A/Ws7wuwAJdCFaUF3fxGe78lmVnsu3mcU02TURAV5c3T+SiQMjGZ4Y0vbNMueirgLKj0B5ljFVHDF63Py4XFt20hsU+EUY4R4QbXS79G82+UUaF3B9QuRM34lJoAtxFhU1jazbV8CnO/P5KqOIukY7QT4ejE+KYMKASC7rFdp2F1TbSsNRqMw1gr4iByqyoTLbmFflQ1We8Y/CySzu4BMKvmHGAGbH5o7XPt0c20ON19ZA+QegE5FAF+Ic1DbY+CqjiDW78vliTwFVdU14uVsY1SuUcUnhXNkvnKhAJ+mS2FhrhHt1gSPk86E632jqOVpsPBnqaJHxuqG65c+wuB8PeZ8Q4+Yq72CwOubewcY6a5AR/tZA8AoAawC4e3Xs7+0CJNCFOE8NTXY2HSxh7Z5C1u4t4EipcRfpgOgAxiVFMK5fOMkxgR3X7t6eGmocAV9s3EFb0yz0a0rgaIkxrys3xsipLTveTfN03K3Hw90rwOiy6RUAnn7g5e9Y9jcGT/PwPmnyMd7v4QMe1mbL3mDpZH8tdSAJdCHagNaazMJqvthTyNo9BWzLKsOuoZuvJ5f1DuXy3mGM7h1KeIDV7FI7TmOtEew/Bnx9JdRVGk099RXG6/ofl6uhvsqYGqqOv7Y3nfv3WjyOB727Fdw8jb8Gjs09jAeOH1tnNS4Y/7jOvdk2i7tjcjv+WrmduM7No4V9LIByNEcpY1lxfJ3WgIZjEauPr/OPNC5unwcJdCHaQenRBr7KKCQ1o5ivfyiiuLoBgH6R/lzeO5TRfcIYlhBy+nHchRFwTfVGsDfVQmMdNNYYz5FtrDlpudaYmm/78T22BmNqqjf+arA1Ol471v24vqnO6EbaVEezpO14ox6Gq35/Xm+VQBeindntmj35lXz9gxHuWw6W0WCz4+luISU+mJE9uzGiZyiDYgPN7TkjDFobfxk01Rs3fdltxvIJ08nrmi3bGo1HIB47A3ecfR9bp5uduTvmcPx1cAKE9jqv0iXQhehgNQ1NbDpQyreZxazfX8LuvEoAfD3dGJ4YwqheoYzo2Y2kyADXaH8XHeZMge7e0cUI0RX4eLoztl84Y/uFA0bzzMYDJazfX8z6zBK+3LcHgACrO8MSQhieGMKwxBCSY+QMXpw/CXQhOkCIryfXJkdxbXIUAHkVtWzYX8Lmg6VsPlTK2r2FAHh7uHFxfBDDE7oxLCGYwd2D8PWS/01F60iTixCdQFFVPVsOlRoBf7CUPfmVaA0WBUlRAQyND2ZofDAXxwUTG+yNkht9uixpQxfCyVTUNvJdVhnbDpexNauM77LKqWkwRmwM9/diaHwwF8UFMaR7MMkxgZ3vLlbRbqQNXQgnE+jtwRV9w7mir9EG32Szs6+gygh4R8h/sjMfADeLom+EvyPgg7goLogeoX5ysbULkjN0IZxUcXU96VnlpB8xpu1HyqmqN27S8fNyZ0B0AINiAxkYE8ig2CDiQ3wk5F2AnKEL4YJC/bwY3z+C8f0jAKMv/P6iar47Us6O7Aq+z6ng3Q2HaWiyA+Dv5c7AmECSYwMZEB3AgOgAEkP9cJOQdxlyhi6EC2u02ckoqGJnTgXfZ1ewM6eCPXlVNNiMkPf2cKNflL8j4AMZGB1I7wg/ubu1E5OLokKIYxptdjILq9mVW8mu3Ap25VSyO6+SakdzjZtFkRjqS79If5KiAkiK8qdfZABRgVbpXdMJSKALIc7IbtdkldawK7eSvfmV7MmrYm9+Jdlltcf2CbC60y8ygD6RfvSNDKBvhD99IvwI8pGnJHWkC25DV0pNBF4F3IB3tNYvnGa/YcBGYIrWevl51iuE6GAWiyIh1JeEUF+uGxR1bH1lXSMZ+VXsya9iT14lGflVrErPpaou69g+EQFe9Inwp2+EP70j/OgV7kevMH8CfTzM+Cld2lkDXSnlBswDrgKygS1KqdVa690t7PdHYE17FCqE6HgBVg9SEkJISQg5tk5rTX5lHfvyq4ypoIqMgire23iYescFWIAwfy96hfk1C3k/eoT5ERHgJU037aQ1Z+jDgUyt9QEApdQSYDKw+6T95gArgGFtWqEQolNRShEV6E1UoPexfvIANrsmu6yGzMJqMgur+cEx//e2nGPdKcEYoKxnuB89Qn3pGeZnvA7zJaGbr1yMvUCtCfQY4Eiz5WzgkuY7KKVigBuBKzlDoCul7gHuAYiLO7/B3YUQnZObRRHfzZf4br6MS4o4tl5rTUFlPQeKqtlfVM3+oqPsL6pmy6EyPkzPPeEzogKtJDqafhK7OeahPnQP8cHLXcL+bFoT6C39bXTyldS/AI9prW1n+lNKa/0W8BYYF0VbW6QQwnkppYgMtBIZaGVkr9ATttU0NHGg6CgHio9yyDEdKD7KxzvyKK9pbPYZEB3oTUKoD/HdfEno5kNciK+xHOIrQx84tCbQs4HuzZZjgdyT9kkBljjCPBS4VinVpLX+sE2qFEK4JB9P42angTGBp2wrr2ngYPFRDpUc5VBxDYdLjnKopIZPduRR1izswbjJKi7Em7gQ42y+e4jPsdeRAdYuc/NUawJ9C9BbKZUI5ABTgWnNd9BaJ/74Wim1EPivhLkQ4kIE+XhyUZwnF8UFn7KtoqaRw6VGwGeVHOVIaS1ZpTVsOVTG6u252Jv9/e/hpogO8iY22JvYIB9jHuJN92AfYoK9Cfd3ncA/a6BrrZuUUg9g9F5xAxZorXcppe51bJ/fzjUKIcQJAn08GOQTxKDYoFO2Ndrs5JYbAZ9VWkN2Wa1jqmHdvkKKqupP2N/dYjQJxQR5G1OwMY8O8iY6yEpUoLfTjEkvNxYJIbqUukYbOeW1HCmtIae8lpyyWnLLa8kpryW3vI68itoTzvDBuKkqOsibqEArkYHeRAdaiTq2bCUq0IqPZ8eEvgzOJYQQDlYPN6O7ZJhfi9ubbHbyK+uOhfvJ8+3ZFZQebTjlfYHeHicEfESAMUX+OA+0Euzj0a598CXQhRCiGXc3C7HBPsQG+5x2n7pGG/kVdeRV1JFfWWvMK4x/BPIra9mZU0Fx9amh7+luISLAiztHJPCzy3u0fe1t/olCCOHirB5ux4ZKOJ2GJjtF1fXkV9RRUGlM+ZV1FFTUEebv1S51SaALIUQ78HS3HLvQ2lEsHfZNQggh2pUEuhBCuAgJdCGEcBES6EII4SIk0IUQwkVIoAshhIuQQBdCCBchgS6EEC7CtMG5lFJFwOHzfHsoUNyG5bgyOVatI8epdeQ4tU57Hqd4rXVYSxtMC/QLoZRKO91oY+JEcqxaR45T68hxah2zjpM0uQghhIuQQBdCCBfhrIH+ltkFOBE5Vq0jx6l15Di1jinHySnb0IUQQpzKWc/QhRBCnEQCXQghXITTBbpSaqJSap9SKlMp9Ruz6+kslFILlFKFSqmdzdaFKKU+V0r94JgHm1ljZ6CU6q6U+lIptUcptUsp9ZBjvRyrZpRSVqXUZqXUdsdx+r1jvRynFiil3JRS3yml/utYNuU4OVWgK6XcgHnANUB/4HalVH9zq+o0FgITT1r3G2Ct1ro3sNax3NU1Ab/SWicBlwL3O/4bkmN1onrgSq31YGAIMFEpdSlynE7nIWBPs2VTjpNTBTowHMjUWh/QWjcAS4DJJtfUKWitU4HSk1ZPBt51vH4X+EmHFtUJaa3ztNbbHK+rMP4njEGO1Qm0odqx6OGYNHKcTqGUigWuA95pttqU4+RsgR4DHGm2nO1YJ1oWobXOAyPIgHCT6+lUlFIJwEXAJuRYncLRjJAOFAKfa63lOLXsL8CvAXuzdaYcJ2cLdNXCOul3Kc6ZUsoPWAE8rLWuNLuezkhrbdNaDwFigeFKqYFm19TZKKWuBwq11lvNrgWcL9Czge7NlmOBXJNqcQYFSqkoAMe80OR6OgWllAdGmC/WWq90rJZjdRpa63LgfxjXaOQ4nWgUMEkpdQijCfhKpdT7mHScnC3QtwC9lVKJSilPYCqw2uSaOrPVwJ2O13cCq0yspVNQSing78AerfWfm22SY9WMUipMKRXkeO0NjAf2IsfpBFrr32qtY7XWCRh5tE5rfQcmHSenu1NUKXUtRpuVG7BAa/2cySV1CkqpfwJXYAzbWQD8DvgQWAbEAVnArVrrky+cdilKqcuAr4EdHG/zfByjHV2OlYNSahDGxTw3jBO/ZVrrPyiluiHHqUVKqSuAR7XW15t1nJwu0IUQQrTM2ZpchBBCnIYEuhBCuAgJdCGEcBES6EII4SIk0IUQwkVIoAshhIuQQBdCCBfx/wGdrXzm99Ds1AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU1f3H8ffJZN/3HUgI+45GwAVlEaWK4obQWmu1SlFxwS76s63aVltr1WqrhaICWlcqUBEVZJGliEoQRCAEkrAkhOw72WfO7487hBASMkDCnZl8X89znztz587MNzf4yfHcc89VWmuEEEK4Pg+zCxBCCNE5JNCFEMJNSKALIYSbkEAXQgg3IYEuhBBuwtOsL46MjNRJSUlmfb0QQrikbdu2FWuto9p6zbRAT0pKIi0tzayvF0IIl6SUOtTea9LlIoQQbkICXQgh3IQEuhBCuAnT+tDb0tjYSG5uLnV1dWaXIgBfX18SExPx8vIyuxQhhAOcKtBzc3MJCgoiKSkJpZTZ5XRrWmtKSkrIzc0lOTnZ7HKEEA5wqi6Xuro6IiIiJMydgFKKiIgI+b8lIVyIUwU6IGHuROR3IYRrcaouFyGEcCdaayprm8ivrCO/so6CijoKKusY0TOUsX3bvDbonEigCyHEGWq02iipbqC4up6iqnqKjq+r6imurqewqp6CSiO86xptp7z/3nEpEujupKmpCU9POfxCOIP6JivF1Q0U2wPZWBooO9ZAac3xdSNlxxooq2mgqq6pzc8J8vEkMsiHqEAfhiWGEhPkQ2yILzHBvsSG+BIb7EtUkA++XpYu+TkkUdpwww03kJOTQ11dHQ899BAzZ85k5cqVPP7441itViIjI1m7di3V1dU88MADpKWloZTiySef5OabbyYwMJDq6moAPvzwQ1asWMGiRYv46U9/Snh4ONu3b+eCCy5g+vTpPPzww9TW1uLn58fChQvp378/VquVRx99lFWrVqGU4p577mHQoEG88sorLFu2DIDVq1czd+5cli5dauahEsIpaa0pq2kkr7yW4up6ymoaKKk2wrj02ImlpLqBour6dgM6wNtCqL834QHehAV4kxThT5i/N2H+3kQEehMV5GMsgT5EBvrg5901Qe0opw3033+8mz15lZ36mYPig3nyusEd7rdgwQLCw8Opra3loosuYurUqdxzzz1s3LiR5ORkSktLAfjjH/9ISEgI33//PQBlZWUdfva+fftYs2YNFouFyspKNm7ciKenJ2vWrOHxxx9nyZIlzJ8/nwMHDrB9+3Y8PT0pLS0lLCyM+++/n6KiIqKioli4cCF33nnnuR0QIVyUzaYprKrncGkNuWU1HCmrJa+iltyyWvLKa8krr6O20XrK+yweijB/b8IDvAgP8GZgXDBjA72JDDSCOTLQh8ggHyLt27qqJd1VnDbQzfT3v/+9uSWck5PD/Pnzufzyy5vHY4eHhwOwZs0a3n///eb3hYWFdfjZ06ZNw2Ix/pFUVFRwxx13sH//fpRSNDY2Nn/urFmzmrtkjn/f7bffzttvv82dd97Jli1beOuttzrpJxbCeWitqa5vsvdHG/3UR8pqOVxaw+HSGnLKasgtq6Wh6eS+6chAbxJC/egXE8S4/tEkhPoRH2p0cYT5exMR4EOQryceHu47estpA92RlnRXWL9+PWvWrGHLli34+/szbtw4hg8fTkZGxin7aq3bHNrXclvrcdwBAQHNj3/3u98xfvx4li1bxsGDBxk3btxpP/fOO+/kuuuuw9fXl2nTpkkfvHA5NpumqLqeI+W1Rqu6vJYj9hb18ROMxdX11DedeiIx2NeTnhH+9I8JYtLAGHqE+xtLmB/xoX4u15ruCpIIrVRUVBAWFoa/vz979+7lq6++or6+ng0bNnDgwIHmLpfw8HCuuuoqXnnlFV566SXA6HIJCwsjJiaG9PR0+vfvz7JlywgKCmr3uxISEgBYtGhR8/arrrqKefPmMW7cuOYul/DwcOLj44mPj+fpp59m9erVXX4shDgbNQ1NHCqpsS/HOFhSw+HSY+SU1nK0opZGqz5p/2BfT+JD/YgK8iE5MsDe9XFyN0h8iB8h/jIFRUck0FuZPHky8+bNY9iwYfTv358xY8YQFRXF/Pnzuemmm7DZbERHR7N69Wp++9vfcv/99zNkyBAsFgtPPvkkN910E88++yxTpkyhR48eDBkypPkEaWu//vWvueOOO3jxxReZMGFC8/a7776bffv2MWzYMLy8vLjnnnuYPXs2ALfddhtFRUUMGjTovBwPIVrTWlNk779uueSUGiFeWFV/0v5h/l70ighgeI9QrhkaR0KYHwmhviSE+hMf6kuQrwR1Z1Fa64736gKpqam69Q0u0tPTGThwoCn1uIrZs2czcuRIfvazn52X75PfSfdltWkOlRwjI7+KvflV7M2vJLvoGDllNaeMrY4L8aVHmD+9IvxJigygZ7g/SREB9IzwJ8RPArszKaW2aa1T23pNWugu5MILLyQgIIAXXnjB7FKEG7HZNHkVtWQWVpNZWE1GfhUZBVXsK6hqDm6lIDkigJToQK7oF0XPCKP/ume4PwnSf+00JNBdyLZt28wuQbiwJquNgyXH2FdgBHdWkbHOLjp20hC/yEBv+scG8aNRvRgQF8SA2CD6RgeZPsZadEwCXQg3o7XmaEVdc1fJvgJjnVVYTYP1RFdJYpgfKVGBjOkdQUpUIH2iA0mJCiAi0MfE6sW5kEAXwkXZbJrcsloyi6rYb2917y+sJquwmqr6E1c+xoX40j82iMv7RtI/Noh+MUGkRAVKi9sNSaAL4eSOjypJz68iI7+SvUeNFnd2cfVJJyejgnzoExXIDSMT6BcbRP8YY5Hhft2HBLoQTqShyca+gir25FWS3hzelZTVNDbvExtstLgvSYmgb4zRVdInSoJbSKALYZrq+ib25FWyO6/Cvq5kf2FV84U3fl4W+scGcfXgWAbEBjEgLpgBsUGE+nubXLlwVhLo56DlrIpCnE5do5U9RyvZmVPOztwKvsstJ7v4GMcvA4kM9GZQfAhX9I9iUFwwg+ODSYoIcOt5R0Tnk0B3AzK3unOx2TSZRdXsOFzOjtxyduaWs/doFU02I72jg4y5sm8YkcCQhBAGxQcTHeQjt/wT58yhFFBKTQZeBizA61rrZ1u9HgK8DfS0f+bzWuuF51TZZ49B/vfn9BGniB0KP3i23ZcfffRRevXqxX333QfAU089hVKKjRs3UlZWRmNjI08//TRTp07t8Kuqq6uZOnVqm+976623eP7551FKMWzYMP79739TUFDArFmzyM7OBmDu3LnEx8czZcoUdu3aBcDzzz9PdXU1Tz31FOPGjeOSSy5h8+bNXH/99fTr14+nn36ahoYGIiIieOedd4iJiWlzzvby8nJ27drF3/72NwBee+010tPTefHFF8/p8HZXhVV1RnjnGMvO3Aqq7aNMgn09GZYYyszLezO8RyjDE0OJDfE1ueJuqLEWakqhtvTktbURvPzAyx+8/e2PA+xrP7BZwdYI1gawNtkfH1/qobEOGmugqc74jpbrpjpoqm+x1J1YD58Bo+7p9B+zw0BXSlmAV4FJQC6wVSm1XGu9p8Vu9wN7tNbXKaWigAyl1Dta64ZOr7gLzZgxg4cffrg50BcvXszKlSuZM2cOwcHBFBcXM2bMGK6//voOW1O+vr4sW7bslPft2bOHZ555hs2bNxMZGdk8t/qDDz7IFVdcwbJly7BarVRXV3c4v3p5eTkbNmwAjInBvvrqK5RSvP766zz33HO88MILbc7Z7u3tzbBhw3juuefw8vJi4cKF/Otf/zrXw9dtFFbVsSWrhM2ZxXyZVUJuWS0Anh6KgXHB3DgygRE9QhnRM5Rk6TZpX2OdEay1ZVBXaYSlzQraCjabfd1kbGuohtpyqCs/dd1w7MT7tM3+WNvfb4W6CmiqPX8/l8UbPH1bLD72tX27b7Cx7gKOtNBHAZla62wApdT7wFSgZaBrIEgZKRcIlAJt3wLEUadpSXeVkSNHUlhYSF5eHkVFRYSFhREXF8ecOXPYuHEjHh4eHDlyhIKCAmJjY0/7WVprHn/88VPet27dOm655RYiIyOBE3Odr1u3rnl+c4vFQkhISIeBPn369ObHubm5TJ8+naNHj9LQ0NA8d3t7c7ZPmDCBFStWMHDgQBobGxk6dOgZHq3uo7Kuka+zS+0BXsy+AuO8SbCvJxenRPDTS5IY2TOUwfEh3esSeGujEZa1ZSfCta4C6iuhvspY6o4/rjyxb409xM8qZBX4hoBfKPiGGuvAGPCwgLKA8mj12AN8gsE/HPzCT11bvI06GmqMlnZjLTQes6/rjPdbvMHDCyye9rWXsfb0MVrxnr5GC9/LHuAe5v0bcCTQE4CcFs9zgdGt9nkFWA7kAUHAdK31KRMaK6VmAjMBevbseTb1drlbbrmFDz/8kPz8fGbMmME777xDUVER27Ztw8vLi6SkpFPmOG9Le+9rb67ztnh6emKznTiMp5tb/YEHHuCRRx7h+uuvZ/369Tz11FNA+3Or33333fzpT39iwIABcuejVpqsNr7LLWfjvmL+l1nMjpxyrDaNr5cHFyWFc+PIRC7tE8Hg+BAsrt76ttmgpgSqC+xLoT2QW4RyfXWLxxVGeNeWQ0PV6T9beYBPkBGoPsFGyzS0F8SPAL+wFku48ZqH14kw9rCc/Ng7wAhwn2AjZEWbHAn0tv7Ftp6i8WpgBzABSAFWK6U2aa1Puoec1no+MB+M2RbPvNyuN2PGDO655x6Ki4vZsGEDixcvJjo6Gi8vL7744gsOHTrk0OdUVFS0+b6JEydy4403MmfOHCIiIprnOp84cSJz587l4Ycfxmq1cuzYMWJiYigsLKSkpITAwEBWrFjB5MmT2/2+43Orv/nmm83b25uzffTo0eTk5PDtt9+yc+fOczlkbuFwSQ0b9xexaX8RX2aVUFXXhFIwLDGUe69I4dI+kVzQKxQfTxdqgTc1QFUeVOTalxxjXZl3IryrC42uibZYfMAn0B7KQeAdBMEJEDPE3jo+Hsj2x76hRjD7BNv3DzBm9RLnjSOBngv0aPE8EaMl3tKdwLPamIs3Uyl1ABgAfNMpVZ5HgwcPpqqqioSEBOLi4rjtttu47rrrSE1NZcSIEQwYMMChz2nvfYMHD+Y3v/kNV1xxBRaLhZEjR7Jo0SJefvllZs6cyRtvvIHFYmHu3LlcfPHFPPHEE4wePZrk5OTTfvdTTz3FtGnTSEhIYMyYMRw4cACg3TnbAW699VZ27Njh0K3z3E1do5VvDpSybm8h6zMKOVhSA0BCqB9ThsUxtm8Ul6REOP+Y75pSKD0ApdnGUmZ/XH4YqvI5pe3lHwnB8RAUawwSCIyBwFgIjLY/jra3hAONLgXhUjqcD10p5QnsAyYCR4CtwI+01rtb7DMXKNBaP6WUigG+BYZrrYvb+1yZD918U6ZMYc6cOUycOLHdfdzpd5JXXssXGYV8sbeIzZnF1DZa8fH04JKUCK7oF8Xl/aJIjgxwvuGD1iYjqIsyoDjDvt5vBHdd+cn7BidAWDKEJUFIYoulhxHk3v6m/Aii85zTfOha6yal1GxgFcawxQVa691KqVn21+cBfwQWKaW+x+iiefR0YS7MVV5ezqhRoxg+fPhpw9zVaa3Zm1/FZ7vy+Xx3PnvzjT7fxDA/pqUmMr5/NBenRDjPiUxrE5RmQeEeKNgDRXuheB+UZBkjQI4LiofIvjDkZgjvfWIJ62WcpBPdlkPj0LXWnwKftto2r8XjPOCqzi3NNXz//ffcfvvtJ23z8fHh66+/NqmijoWGhrJv3z6zy+gSWmt25lbw2a58Vu46ysGSGjwUpCaF8/g1A5gwIJqUqEDzWuE2Kxwrhup8oy+7aC8UphsBXpxhjHcG44RiWDJE9Yd+k411ZH8jyH2DzaldOD2nu7zwTEaBOIOhQ4eyY8cOs8voEmbdnvBMaa3ZnlPOJzuPsnJXPkfKa7F4KC5JiWDm5SlcNTiGyPMxx7fNZgR1+WH7csg4CVlVYGyvym/7JGRwIkQPhD4TIHqQ8Tiyn7S2xRlzqkD39fWlpKSEiIgIlwp1d6S1pqSkBF9f572qMbOwio925PHRjjwOl9bgbfFgbN9IHr6yL5MGxXTNCc2meig7CCWZ9iXLCO7yw1Cec3LXCIB/BATFGSchowcb6+Ylzt7iDun8OkW35FSBnpiYSG5uLkVFRWaXIjD+wCYmJppdxknyK+pY/t0RPtqRx+68SjwUXNonkgcn9uWqwTEEd9Yd5LU2AjvnGzj63YkAr8gxrkY8zj8SwpMhbgQMvB5CexpjrUN7Gicj5SSkOI+cKtC9vLyar3AU4rj6Jisrd+XzwdYctmSXoDUMTwzhiSmDmDI8juigTvi/iLpKOLINcreeWGrtV+p6BUBkH0hMNebgiOgDESkQnmKMwRbCSThVoAvRUnZRNe99c5gPt+VSVtNIz3B/HprYl6kjEkiODOj4A06nqQFyvoLMtZC1zj4RnAYURA2AAVOgxyhIvMg4GSlXJwoXIIEunEpDk41Vu/N59+vDbMkuwdNDMWlQDD8a3ZNLUyLPfqIrrY1x25lrIWstHNhkzNnh4Qk9xsC4x4zwTrhQWt3CZUmgC6dQWFXHW18e4r1vDlNyrIHEMD9+dXV/pqUmnn2XSvlhOLgZDv3PCPBy+7QNYckw4oeQMhGSxxqXqQvhBiTQhan2F1Tx+qYDLNt+hEabjYkDYvjxmJ5c3jfqzFrjWhuBffB/J0K8/LDxmm8o9LoELnkAUiYY/d9CuCEJdHHeaa3ZklXCa5uy+SKjCF8vD6Zf1IO7Lkt2vG+8tgzythsnMo/Y19X5xmt+4UaAj7kfki41hgtKH7joBiTQxXljtWlW7Mxj/sZsdudVEhnozSOT+vHjMb0IDzjNmHGtjblLsr+A3DQjvEuzTrwe0QeSLzdOYva61DipKQEuuiEJdNHltNasSS/k+VUZZBRUkRIVwLM3DeWGkQntz6NSVwEHNkLmGuNEZoV9Sv6geEi4AEbeBvEXQPxIOYkphJ0EuuhSX2YV89dVGWw/XE5yZAD/+OFIrh0a13b/eFEGpH9sBHjO18Yl8t5B0PsKGPuIcRIzrNf5/yGEcBES6KJL7Mwt56+rMti0v5jYYF/+fNNQbrkwES9Lq66Q6kL4/kPY+QEctc+JEzccLn0I+lxpdKNYOunqTyHcnAS66FSZhVW88Pk+PtuVT5i/F7+9diA/HtPr5K6VhhrY+4kR4lnrjJZ43HC4+s8w5CZjnhMhxBmTQBedIreshpfW7Gfpt7n4eVl4aGJf7h6bTNDxuVWsjZC9AXZ9aHSrNFQbswxe+hAMmw7Rjt0JSgjRPgl0cU6Kqup59YtM3vn6EEop7ro0mXvHpRAR6GPM/X1gE+xaAns+gtpS8AmBwTfAsBnGiBQZjSJEp5FAF2eloraR+RuzWPC/gzRYbdyamsgDE/oSH+JrDC3ctAR2LzPGhnv5Q/9rjDvs9Jko96oUootIoIsz0tBkY9GXB3j1iywqahu5bng8c67sS2+vUvjuH/Dde8acKRYf6DvJCPF+Vxt3gBdCdCkJdOGwL7OKeeKj3WQWVjOufxSPTkhkYNl6+OR3cHCTsVPSWBj7Sxg4RW7cIMR5JoEuOlRYWcfTn6Sz/Ls8eoT78eG1HqQWL4C3lxszFoYlw/jfGCc3ZZy4EKaRQBftarLaeHPLIf62eh8NVhvPjGpgRvXrWNauA59gGHozjLgNeowGuWWgEKaTQBdtSjtYym//u4u9+VXcllTNb/yW4L9zlTHx1aQ/wkV3y+3VhHAyEujiJHWNVp5bmcGCzQcYHVTKlr6fEpvzGconyOhWGXOvzB8uhJOSQBfN9uZX8tB7OygtyGFp/CeMLPsMle8Ll80x5hL3Dze7RCHEaUigC2w2zYLNB/jrynR+4rOBR4PexbOiHkbPgssegcAos0sUQjhAAr2bK6is45f/+Y68zO/4OPhN+tXvgh6XwXUvQWRfs8sTQpwBCfRubOWuozyxZBt3WJcyy/cjPFQgXP8KjPyxjFoRwgU5FOhKqcnAy4AFeF1r/Wyr138F3NbiMwcCUVrr0k6sVXSSukYrv/94N9lbP2ep/0ISdS4MmWbMdijdK0K4rA4DXSllAV4FJgG5wFal1HKt9Z7j+2it/wr81b7/dcAcCXPnlFNawwNvf8UNhXP5s8/n6KCecO0S6Hul2aUJIc6RIy30UUCm1jobQCn1PjAV2NPO/j8E3uuc8kRnWre3gGfeX8cLvMgIz30w+l7UxN/JPCtCuAlHAj0ByGnxPBcY3daOSil/YDIwu53XZwIzAXr27HlGhYqzZ7Vp/rZ6H1+tX8F/fP9BqGc9TF1o3ExCCOE2HAn0ts6O6Xb2vQ7Y3F53i9Z6PjAfIDU1tb3PEJ2opLqeB9/7lpSD7/GBz9uosCQ8ZrwD0QPNLk0I0ckcCfRcoEeL54lAXjv7zkC6W5zGtkNlPPL2FubU/5MbvDZBvx/AjfPAL9Ts0oQQXcCRQN8K9FVKJQNHMEL7R613UkqFAFcAP+7UCsVZWZyWw9xla3nN+yX6ehw0Ltsf+0u5Q5AQbqzDQNdaNymlZgOrMIYtLtBa71ZKzbK/Ps++643A51rrY11WreiQzaZ5blUGezYtZbnvPwn08kDdvBj6XWV2aUKILqa0NqcrOzU1VaelpZny3e6qtsHKI+9/S0rGv3jE60NU9CDUjLchvLfZpQkhOolSapvWOrWt1+RKUTdRWFnHQ4vWc3fRs0z02o4editqyssyxa0Q3YgEuhvYk1fJnxb+h780/IVEr1KY/Dzqorvl8n0huhkJdBe3Nr2Az9/7O697zMcSEIbHDz+FHqPMLksIYQIJdBf23pZMGj99jL9YVtOQeAleM96EwGizyxJCmEQC3UUt3rCd3mtnMdqyl8bR9+N91R/AIr9OIbozSQAX9NHqdYze9HPiLWU0Tn0NrxG3ml2SEMIJSKC7mNUfv8v4tF+gvXzg9k/wSmpzWh0hRDcklw26kK/f/zPj0+6nwicWv/s2SJgLIU4iLXRXYG0ifdF9jM75gO8CLmbg/R/gHRBidlVCCCcjge7sasvJfW0GA0u38HnIrYy7/594e3uZXZUQwglJl4szKz9M2SvjiSn5hreifsn4B/8lYS6EaJe00J1V/vfULLwRS90x/pHwHA/87C68LPL3VwjRPgl0Z5S9nsZ3f0R5ow/zEl/mt3feImEuhOiQBLqz2bkY27L7yLbG8mrCszz30x/g7SlhLoTomAS6s9AaNr8Ma57kG9sg5sX9gbl3TsDXy2J2ZUIIFyGB7gxsVlj5f/DNv/jEdjELoh/jzbsuw89bwlwI4TgJdLM11sHSuyH9YxbYrmVJ+M95965LCPSRX40Q4sxIapipqR4+uA0y1/AX209YHXoLH9w9hhB/GZoohDhzEuhmaWqAxXdA5hqeYibrg65h8d2jiQj0MbsyIYSLkkA3g7URPrwT9n3G854zWekxmSX3jCE62NfsyoQQLkzGw51v1iZYOhP2rmBh4Exer5/Aaz9JJSHUz+zKhBAuTgL9fLJZ4aP7YPdSVsTcy++Lx/HCtBEMTZSJtoQQ504C/Xyx2WD5g7DzA7b2vp/Zh8by8JV9uXZYnNmVCSHchAT6+aA1fDIHdrzNgcGzuTX9Uq4dGseDE/qaXZkQwo1IoJ8Pn/8Wti2iZORsrts1lsHxwTw/bTgeHsrsyoQQbkRGuXS17z+ELa9QN/Jn3Lh3In7eNl77SapcBSqE6HTSQu9KRRmw/EFsPcZw19Ebya+qZ/7tFxIXIiNahBCdz6FAV0pNVkplKKUylVKPtbPPOKXUDqXUbqXUhs4t0wU1HIPFPwEvP54PeowvD1by3M3DGNkzzOzKhBBuqsMuF6WUBXgVmATkAluVUsu11nta7BMK/BOYrLU+rJSK7qqCXYLWsGIOFGWwY9xC/rmyhnvGJnPDyASzKxNCuDFHWuijgEytdbbWugF4H5jaap8fAUu11ocBtNaFnVumi9m2CHZ+QN1lj/LzL4PoFxPIL6/ub3ZVQgg350igJwA5LZ7n2re11A8IU0qtV0ptU0r9pK0PUkrNVEqlKaXSioqKzq5iZ5e3Az57FFIm8njx1RRXN/DCtBH4eMpJUCFE13Ik0NsaW6dbPfcELgSuBa4GfqeU6nfKm7Ser7VO1VqnRkVFnXGxTq+2HP5zBwREsm7w0yzdcZT7x/eRK0GFEOeFI8MWc4EeLZ4nAnlt7FOstT4GHFNKbQSGA/s6pUpXoDV8dD9U5FI5Yzm/XnyEQXHBzB7fx+zKhBDdhCMt9K1AX6VUslLKG5gBLG+1z0fAWKWUp1LKHxgNpHduqU5uyyuwdwVc9TT/l+ZHRW0jL04fLvcDFUKcNx220LXWTUqp2cAqwAIs0FrvVkrNsr8+T2udrpRaCewEbMDrWutdXVm4UznyLax+EgZezwq/6/lk5w5+dXV/BsQGm12ZEKIbUVq37g4/P1JTU3VaWpop392prE3w2nioLqT4p/9j0j+30zMigCWzLsbTIq1zIUTnUkpt01qntvWaXPp/rr6ZD/k70dPe5P8+PcSxBisvTBsmYS6EOO8kdc5FRS6sexr6XsWyugtZvaeAX13Vnz7RQWZXJoTohiTQz8Vnj4K2UXz5Mzz18R5Se4Vx12XJZlclhOimJNDP1t5PjFEt4x7jmS9rqGu08ddpw7HIlLhCCJNIoJ+N+ir49FcQPZi0+B+ybPsRZl7em+TIALMrE0J0Y3JS9Gx88WeozMN680Ke+O8+4kN8uW98itlVCSG6OWmhn6mj38HXcyH1Tt49Gsueo5X85tpB+HvL30YhhLkk0M+EzQofPwT+kZRf/DgvfJ7Bxb0juGZorNmVCSGEdLmcka1vQN52uPkNntuYT1VdE7+fOhil5ESoEMJ80kJ3VGUerP0DpExgV9iVvPfNYX5ycS/6xciYcyGEc5BAd9Sq34CtEdsPXuCJ5buJCPDm4StPmSFYCCFMI4HuiENfwu6lcOnDLDvkzbeHy/n15AGE+HmZXZkQQjSTQO+IzWpcERqcSFXqfQGbR74AAA9CSURBVPz5s72M6BHKLRckml2ZEEKcRE6KdmTHO5C/E25+g5c3HKHkWD1v3JGKh1wRKoRwMtJCP526SuNEaI8xZEZfxaIvD3LrhT0Y3iPU7MqEEOIU0kI/nY1/hWPFcNt/eHblPvy8Lfxqcn+zqxJCiDZJC709JVnw1VwYeRsZHn1Yk17Azy5LJjLQx+zKhBCiTRLo7Vn1G/D0hQlPMHd9Jv7eFn56SZLZVQkhRLsk0NuSuRb2fQaX/5KcxiA+3nmUH43qSai/t9mVCSFEuyTQW7M2warHISwZxtzLvzZm4aHg7rG9za5MCCFOS06Ktpa2AIr2wox3KaqFxWm53DQykdgQX7MrE0KI05IWeks1pfDFM5B8BfS/hgWbD9BotfHzK6R1LoRwfhLoLX3xJ6ivhMnPUlnfxNtbDnHNkDh6RwWaXZkQQnRIAv24kiyjuyX1LogZxL+3HKKqvol7x8mdiIQQrkEC/bjNL4OHJ1z+K+oarSzcfIDL+0UxJCHE7MqEEMIhEugAFUdgx7sw8scQFMvitByKqxu4T1rnQggXIoEOsOUV0Da49CEarTb+tSGbC3qGMjo53OzKhBDCYRLox4ohbSEMmw5hvfj4uzyOlNdy37g+cms5IYRLcSjQlVKTlVIZSqlMpdRjbbw+TilVoZTaYV+e6PxSu8hXc6GpDi6bg82mmbs+i/4xQUwYEG12ZUIIcUY6vLBIKWUBXgUmAbnAVqXUcq31nla7btJaT+mCGrtOXQV88xoMuh6i+rF2TwH7C6t5afoIme9cCOFyHGmhjwIytdbZWusG4H1gateWdZ5sfR3qK2DsL9Ba88/1mfQI92PKsDizKxNCiDPmSKAnADktnufat7V2sVLqO6XUZ0qpwW19kFJqplIqTSmVVlRUdBbldqKGGtjyT+gzCeKGszO3gu2Hy7n7st54WuTUghDC9TiSXG31PehWz78FemmthwP/AP7b1gdpredrrVO11qlRUVFnVmln+/YtqCmGsb8A4IO0HHy9PLjxgrb+VgkhhPNzJNBzgR4tnicCeS130FpXaq2r7Y8/BbyUUpGdVmVna2qAL/8OvS6FXhdT09DE8h15XDM0jmBfL7OrE0KIs+JIoG8F+iqlkpVS3sAMYHnLHZRSsco+xk8pNcr+uSWdXWyn2fk+VB6BsY8A8On3+VTXNzHjop4mFyaEEGevw1EuWusmpdRsYBVgARZorXcrpWbZX58H3ALcq5RqAmqBGVrr1t0yzsHaBP/7G8SNgJSJAHyw9TC9IwO4KCnM5OKEEOLsOTQfur0b5dNW2+a1ePwK8ErnltZF9vwXSrPh1n+DUmQVVbP1YBmP/WCAXEgkhHBp3Ws4h80Gm16EyP4wwBgyv3hrDhYPxU1yMlQI4eK6V6Dv/xwKdxt95x4eNFptLPk2l4kDookOkjsSCSFcW/cK9O3/hsBYGHILAGvTCymubmD6RT06eKMQQji/7hPodRVGC33wjWAxTh0sTsshJtiHK/qZPCZeCCE6QfcJ9L2fgrUBhtwMwNGKWtZnFHLLhYlyZagQwi10nyTbtQRCekJiKgAfpuVi03BrqnS3CCHcQ/cI9JpSyP4ChtwISmGzaRZvy+GSlAh6RQSYXZ0QQnSK7hHo6cvB1gSDbwJgS3YJOaW1cjJUCOFWukeg71oK4SkQNxyAD7bmEOLnxdWDY00uTAghOo/7B3pVARzcZJwMVYrymgZW7s7nhhHx+HpZzK5OCCE6jfsH+p6PjBtADzG6W/67/QgNTTamy0RcQgg34/6BvnspRA+C6IForXl/aw5DE0IYFB9sdmVCCNGp3DvQK3Lh8Jbmk6HfH6lgb36VnAwVQrgl9w703fYbJ9m7W5ZtP4K3pwfXj4g3sSghhOga7h3ou5YY855HpKC1Zk16AZf1iZS7Egkh3JL7BnppNuR929w631dQTU5pLZMGxZhcmBBCdA33DfTdy4z14BsBWL0nH4CJA6LNqkgIIbqU+wb6rqXQYzSEGsMTV6cXMrxHKNHBMu+5EMI9uWegF2VAwa7m0S2FlXV8l1POVdLdIoRwY+4Z6LuWAgoG3wDAmvRCAK4cKIEuhHBf7hfoWhujW5IugyBjrpY16QX0CPejX0ygycUJIUTXcb9AL9gFJfubR7fUNDTxv8xirhwYg1LK5OKEEKLruF+g71oCygIDpwKwcV8xDU02Ga4ohHB77hXoWhtXh/YeBwERgNHdEuzryUVJ4aaWJoQQXc29Ar00G8oOwIBrALDaNOv2FjJ+QDRect9QIYSbc6+Uy1pnrFMmAPDt4TJKjzVId4sQoltwv0APS4Lw3gCs2VOAl0Vxeb8oc+sSQojzwKFAV0pNVkplKKUylVKPnWa/i5RSVqXULZ1XooOaGuDAxubWOcDq9ALG9I6QybiEEN1Ch4GulLIArwI/AAYBP1RKDWpnv78Aqzq7SIfkboWGakiZCEBWUTXZRceku0UI0W040kIfBWRqrbO11g3A+8DUNvZ7AFgCFHZifY7LWmcMV0weCxjdLQAT5epQIUQ34UigJwA5LZ7n2rc1U0olADcC8073QUqpmUqpNKVUWlFR0ZnWenpZ6yDxIvANAYzhioPigkkI9evc7xFCCCflSKC3dXmlbvX8JeBRrbX1dB+ktZ6vtU7VWqdGRXXiicqaUsjb3tx/XlJdz7ZDZdLdIoToVjwd2CcXaHkTzkQgr9U+qcD79kvrI4FrlFJNWuv/dkqVHcn+AtDQx+g/X7e3EJtGAl0I0a04Euhbgb5KqWTgCDAD+FHLHbTWyccfK6UWASvOW5iD0d3iGwLxIwGjuyUuxJfB8cHnrQQhhDBbh10uWusmYDbG6JV0YLHWerdSapZSalZXF9ghrSHrC+Nyfw8LdY1WNu6TybiEEN2PIy10tNafAp+22tbmCVCt9U/PvawzULwPKo9Ayq8B+DKrmNpGK1dKd4sQoptx/StFM9caa/sJ0dV7Cgn08WRMb5mMSwjRvbh+oGetg4i+ENoTm02zNr2Ay/tF4uNpMbsyIYQ4r1w70Jvq4eD/mlvnO49UUFhVL6NbhBDdkmsH+uGvoKm2OdDX7CnA4qEY3z/a5MKEEOL8c+1Az1oHHl7G/UMxhitelBRGqL+3yYUJIcT55+KBvhZ6jgGfQHJKa9ibX8WVMneLEKKbct1Ary6E/O8hZTwAq+2TcUn/uRCiu3LdQM9eb6yP95+nF9A3OpBeEQHm1SSEECZy3UDPWgf+ERA7nIqaRr4+UCqtcyFEt+aaga61Eei9x4OHB+v3FWK1abk6VAjRrblmoBfshuqCFleHFhAZ6M2IxFCTCxNCCPO4ZqBnrTPWKeNpaLKxIaOIiQNi8PCQybiEEN2X6wZ61EAIjuebA6VU1TdJ/7kQottzvUBvrIVDX7bobsnH18uDS/tEmlyYEEKYy/UC/dBmsNZDnwlorVmTXshlfaLw85bJuIQQ3ZvrBXpAFIy8HXpeQvrRKo6U1zJpkMzdIoQQDt3gwqnEDYeprwCwes9+lIIJA6T/XAghXK+F3sKa9AJG9gglKsjH7FKEEMJ0LhvoRytq+f5IhVxMJIQQdi4b6GvTCwGYJLMrCiEE4MKBvnpPAb0i/OkTHWh2KUII4RRcMtCr65vYklXCpIExKCVXhwohBLhooG/aV0SD1Sb950II0YJLBvrqPQWE+HmR2ivM7FKEEMJpuFygN1ltrMsoZMKAaDwtLle+EEJ0GZdLxG2HyiivaZTJuIQQohWXC3SLh+KKflGM7SuTcQkhREsOBbpSarJSKkMplamUeqyN16cqpXYqpXYopdKUUpd1fqmG1KRw3rxrFEG+Xl31FUII4ZI6nMtFKWUBXgUmAbnAVqXUcq31nha7rQWWa621UmoYsBgY0BUFCyGEaJsjLfRRQKbWOltr3QC8D0xtuYPWulprre1PAwCNEEKI88qRQE8Aclo8z7VvO4lS6kal1F7gE+CuzilPCCGEoxwJ9LYuxTylBa61Xqa1HgDcAPyxzQ9Saqa9jz2tqKjozCoVQghxWo4Eei7Qo8XzRCCvvZ211huBFKXUKcNQtNbztdapWuvUqKioMy5WCCFE+xwJ9K1AX6VUslLKG5gBLG+5g1Kqj7JPqqKUugDwBko6u1ghhBDt63CUi9a6SSk1G1gFWIAFWuvdSqlZ9tfnATcDP1FKNQK1wPQWJ0mFEEKcB8qs3E1NTdVpaWmmfLcQQrgqpdQ2rXVqm6+ZFehKqSLg0Fm+PRIo7sRy3JkcK8fIcXKMHCfHdOVx6qW1bvMkpGmBfi6UUmnt/YUSJ5Nj5Rg5To6R4+QYs46Ty83lIoQQom0S6EII4SZcNdDnm12AC5Fj5Rg5To6R4+QYU46TS/ahCyGEOJWrttCFEEK0IoEuhBBuwuUCvaObbXRXSqkFSqlCpdSuFtvClVKrlVL77etuf1dtpVQPpdQXSql0pdRupdRD9u1yrFpQSvkqpb5RSn1nP06/t2+X49QGpZRFKbVdKbXC/tyU4+RSgd7iZhs/AAYBP1RKDTK3KqexCJjcattjwFqtdV+Mm5DIH0BoAn6htR4IjAHut/8bkmN1snpggtZ6ODACmKyUGoMcp/Y8BKS3eG7KcXKpQMeBm210V/ZZLktbbZ4KvGl//CbG1Mbdmtb6qNb6W/vjKoz/CBOQY3USbai2P/WyLxo5TqdQSiUC1wKvt9hsynFytUB36GYbolmM1vooGEEGRJtcj1NRSiUBI4GvkWN1Cns3wg6gEFittZbj1LaXgF8DthbbTDlOrhboDt1sQ4iOKKUCgSXAw1rrSrPrcUZaa6vWegTGPRBGKaWGmF2Ts1FKTQEKtdbbzK4FXC/Qz+hmG4ICpVQcgH1daHI9TkEp5YUR5u9orZfaN8uxaofWuhxYj3GORo7TyS4FrldKHcToAp6glHobk46TqwV6hzfbECdZDtxhf3wH8JGJtTgF+41Y3gDStdYvtnhJjlULSqkopVSo/bEfcCWwFzlOJ9Fa/5/WOlFrnYSRR+u01j/GpOPkcleKKqWuweizOn6zjWdMLskpKKXeA8ZhTNtZADwJ/BdYDPQEDgPTtNatT5x2K0qpy4BNwPec6PN8HKMfXY6VnVJqGMbJPAtGw2+x1voPSqkI5Di1SSk1Dvil1nqKWcfJ5QJdCCFE21yty0UIIUQ7JNCFEMJNSKALIYSbkEAXQgg3IYEuhBBuQgJdCCHchAS6EEK4if8HNeMW5vV8IqMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "history = txtcls_history\n",
    "pd.DataFrame(history.history)[['loss', 'val_loss']].plot()\n",
    "pd.DataFrame(history.history)[['accuracy', 'val_accuracy']].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling predicition from model head produces output from final dense layer. This final layer is used to compute categorical cross-entropy when training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.81729865, 0.02749203, 0.1552093 ]], dtype=float32)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txtcls_model.predict(x=[\"YouTube introduces Video Chapters to make it easier to navigate longer videos\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save the model artifacts in the local directory called `./txtcls_swivel`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: ./txtcls_swivel/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: ./txtcls_swivel/assets\n"
     ]
    }
   ],
   "source": [
    "tf.saved_model.save(txtcls_model, './txtcls_swivel/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "....and examine the model's serving default signature. As expected the model takes as input a text string (e.g. an article title) and retrns a 3-dimensional vector of floats (i.e. the softmax output layer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The given SavedModel SignatureDef contains the following input(s):\n",
      "  inputs['text'] tensor_info:\n",
      "      dtype: DT_STRING\n",
      "      shape: (-1)\n",
      "      name: serving_default_text:0\n",
      "The given SavedModel SignatureDef contains the following output(s):\n",
      "  outputs['outputs'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1, 3)\n",
      "      name: StatefulPartitionedCall_2:0\n",
      "Method name is: tensorflow/serving/predict\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show \\\n",
    " --tag_set serve \\\n",
    " --signature_def serving_default \\\n",
    " --dir ./txtcls_swivel/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify the returned predictions, we'll modify the model signature so that the model outputs the predicted article source (either `nytimes`, `techcrunch`, or `github`) rather than the final softmax layer. We'll also return the 'confidence' of the model's prediction. This will be the softmax value corresonding to the predicted article source."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function(input_signature=[tf.TensorSpec([None], dtype=tf.string)])\n",
    "def source_name(text):\n",
    "    labels = tf.constant(['github', 'nytimes', 'techcrunch'], dtype=tf.string)\n",
    "    probs = txtcls_model(text, training=False)\n",
    "    indices = tf.argmax(probs, axis=1)\n",
    "    pred_source = tf.gather(params=labels, indices=indices)\n",
    "    pred_confidence = tf.reduce_max(probs, axis=1)\n",
    "    \n",
    "    return {'source': pred_source,\n",
    "            'confidence': pred_confidence}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll re-save the new Swivel model that has this updated model signature by referencing the `source_name` function for the model's `serving_default`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: ./txtcls_swivel/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: ./txtcls_swivel/assets\n"
     ]
    }
   ],
   "source": [
    "shutil.rmtree('./txtcls_swivel', ignore_errors=True)\n",
    "txtcls_model.save('./txtcls_swivel', signatures={'serving_default': source_name})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine the model signature to confirm the changes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The given SavedModel SignatureDef contains the following input(s):\n",
      "  inputs['text'] tensor_info:\n",
      "      dtype: DT_STRING\n",
      "      shape: (-1)\n",
      "      name: serving_default_text:0\n",
      "The given SavedModel SignatureDef contains the following output(s):\n",
      "  outputs['confidence'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1)\n",
      "      name: StatefulPartitionedCall_2:0\n",
      "  outputs['source'] tensor_info:\n",
      "      dtype: DT_STRING\n",
      "      shape: (-1)\n",
      "      name: StatefulPartitionedCall_2:1\n",
      "Method name is: tensorflow/serving/predict\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show \\\n",
    " --tag_set serve \\\n",
    " --signature_def serving_default \\\n",
    " --dir ./txtcls_swivel/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now when we call predictions using the updated serving input function, the model will return the predicted article source as a readable string, and the model's confidence for that prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "title1 = \"House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force\"\n",
    "title2 = \"YouTube introduces Video Chapters to make it easier to navigate longer videos\"\n",
    "title3 = \"As facebook turns 10  zuckerberg wants to change how tech industry works\"\n",
    "\n",
    "restored = tf.keras.models.load_model('./txtcls_swivel')\n",
    "infer = restored.signatures['serving_default']\n",
    "outputs = infer(text=tf.constant([title1, title2, title3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[b'github' b'github' b'techcrunch']\n",
      "[0.9840866 0.8172988 0.9124565]\n"
     ]
    }
   ],
   "source": [
    "print(outputs['source'].numpy())\n",
    "print(outputs['confidence'].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the model for online serving\n",
    "\n",
    "Once the model is trained and the assets saved, deploying the model to GCP is straightforward. After some time you should be able to see your deployed model and its version on the [model page of GCP console](https://console.cloud.google.com/ai-platform/models)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using endpoint [https://ml.googleapis.com/]\n",
      "WARNING: Please explicitly specify a region. Using [us-central1] by default on https://ml.googleapis.com. Please note that your model will be inaccessible from https://us-central1-ml.googelapis.com\n",
      "\n",
      "Learn more about regional endpoints and see a list of available regions: https://cloud.google.com/ai-platform/prediction/docs/regional-endpoints\n",
      "Created ml engine model [projects/munn-sandbox/models/txtcls_mm].\n",
      "Using endpoint [https://ml.googleapis.com/]\n",
      "Creating version (this might take a few minutes)......\n",
      "...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "MODEL_NAME=\"txtcls\"\n",
    "MODEL_VERSION=\"swivel\"\n",
    "MODEL_LOCATION=\"./txtcls_swivel/\"\n",
    "\n",
    "gcloud ai-platform models create ${MODEL_NAME}\n",
    "\n",
    "gcloud ai-platform versions create ${MODEL_VERSION} \\\n",
    "--model ${MODEL_NAME} \\\n",
    "--origin ${MODEL_LOCATION} \\\n",
    "--staging-bucket gs://${BUCKET} \\\n",
    "--runtime-version=2.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the Evaluation job on CAIP\n",
    "\n",
    "Now that the model is deployed, go to [Cloud AI Platform](https://console.cloud.google.com/ai-platform/models/txtcls/versions) to see the model version you've deployed and [set up an evaluation job](https://console.cloud.google.com/ai-platform/models/txtcls/versions/swivel/evaluation) by clicking on the button called \"Create Evaluation Job\". You will be asked to provide some relevant information:\n",
    " - Job description: txtcls_swivel_eval\n",
    " - Model objective: text classification\n",
    " - Classification type: single-label classification\n",
    " - Prediction label file path for the annotation specification set: When you create an evaluation job on CAIP, you must specify a CSV file that defines your annotation specification set. This file must have one row for every possible label your model outputs during prediction. Each row should be a comma-separated pair containing the label and a description of the label: label-name,description\n",
    " - Daily sample percentage: We'll set this to 100% so that all online predicitons are captured for evaluation.\n",
    " - BigQuery table to house online prediction requests: We'll use the BQ dataset and table `txtcls_eval.swivel`. If you enter a BigQuery table that doesn’t exist, one with that name will be created with the correct schema. \n",
    " - Prediction input\n",
    "     - Data key: this is The key for the raw prediction data. From examining our deployed model signature, the input data key is `text`.\n",
    "     - Data reference key: this is for image models, so we can ignore\n",
    " - Prediction output\n",
    "     - Prediction labels key: This is the prediction key which contains the predicted label (i.e. the article source). For our model, the label key is `source`.\n",
    "     - Prediction score key: This is the prediction key which contains the predicted scores (i.e. the model confidence). For our model, the score key is `confidence`.\n",
    " - Ground-truth method: Check the box that indicates we will provide our own labels, and not use a Human data labeling service.\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the evaluation job is set up, the table will be made in BigQuery to capture the online prediction requests. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The google.cloud.bigquery extension is already loaded. To reload it, use:\n",
      "  %reload_ext google.cloud.bigquery\n"
     ]
    }
   ],
   "source": [
    "%load_ext google.cloud.bigquery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>model_version</th>\n",
       "      <th>time</th>\n",
       "      <th>raw_data</th>\n",
       "      <th>raw_prediction</th>\n",
       "      <th>groundtruth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [model, model_version, time, raw_data, raw_prediction, groundtruth]\n",
       "Index: []"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "SELECT * FROM `txtcls_eval.swivel`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, every time this model version receives an online prediction request, this information will be captured and stored in the BQ table. Note, this happens everytime because we set the sampling proportion to 100%. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Send prediction requests to your model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some article titles and their groundtruth sources that we can test with prediciton.\n",
    "\n",
    "| title  | groundtruth  |\n",
    "|---|---|\n",
    "| YouTube introduces Video Chapters to make it easier to navigate longer videos  |  techcrunch |\n",
    "| A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison | nytimes |\n",
    "| A native Mac app wrapper for WhatsApp Web | github |\n",
    "| Astronauts Dock With Space Station After Historic SpaceX Launch | nytimes |\n",
    "| House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force | nytimes |\n",
    "| Scrollability | github |\n",
    "| iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks | techcrunch |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting input.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile input.json\n",
    "{\"text\": \"YouTube introduces Video Chapters to make it easier to navigate longer videos\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONFIDENCE  SOURCE\n",
      "0.512727    techcrunch\n"
     ]
    }
   ],
   "source": [
    "!gcloud ai-platform predict \\\n",
    "  --model txtcls \\\n",
    "  --json-instances input.json \\\n",
    "  --version swivel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting input.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile input.json\n",
    "{\"text\": \"A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONFIDENCE  SOURCE\n",
      "0.429436    techcrunch\n"
     ]
    }
   ],
   "source": [
    "!gcloud ai-platform predict \\\n",
    "  --model txtcls \\\n",
    "  --json-instances input.json \\\n",
    "  --version swivel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting input.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile input.json\n",
    "{\"text\": \"A native Mac app wrapper for WhatsApp Web\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONFIDENCE  SOURCE\n",
      "0.482146    techcrunch\n"
     ]
    }
   ],
   "source": [
    "!gcloud ai-platform predict \\\n",
    "  --model txtcls \\\n",
    "  --json-instances input.json \\\n",
    "  --version swivel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting input.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile input.json\n",
    "{\"text\": \"Astronauts Dock With Space Station After Historic SpaceX Launch\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONFIDENCE  SOURCE\n",
      "0.516605    techcrunch\n"
     ]
    }
   ],
   "source": [
    "!gcloud ai-platform predict \\\n",
    "  --model txtcls \\\n",
    "  --json-instances input.json \\\n",
    "  --version swivel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting input.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile input.json\n",
    "{\"text\": \"House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONFIDENCE  SOURCE\n",
      "0.524791    nytimes\n"
     ]
    }
   ],
   "source": [
    "!gcloud ai-platform predict \\\n",
    "  --model txtcls \\\n",
    "  --json-instances input.json \\\n",
    "  --version swivel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting input.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile input.json\n",
    "{\"text\": \"Scrollability\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONFIDENCE  SOURCE\n",
      "0.510411    techcrunch\n"
     ]
    }
   ],
   "source": [
    "!gcloud ai-platform predict \\\n",
    "  --model txtcls \\\n",
    "  --json-instances input.json \\\n",
    "  --version swivel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting input.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile input.json\n",
    "{\"text\": \"iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONFIDENCE  SOURCE\n",
      "0.484371    nytimes\n"
     ]
    }
   ],
   "source": [
    "!gcloud ai-platform predict \\\n",
    "  --model txtcls \\\n",
    "  --json-instances input.json \\\n",
    "  --version swivel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summarizing the results from our model:\n",
    "\n",
    "| title  | groundtruth  | predicted\n",
    "|---|---|---|\n",
    "| YouTube introduces Video Chapters to make it easier to navigate longer videos  |  techcrunch | techcrunch |\n",
    "| A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison | nytimes | techcrunch |\n",
    "| A native Mac app wrapper for WhatsApp Web | github | techcrunch |\n",
    "| Astronauts Dock With Space Station After Historic SpaceX Launch | nytimes | techcrunch |\n",
    "| House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force | nytimes | nytimes |\n",
    "| Scrollability | github | techcrunch |\n",
    "| iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks | techcrunch | nytimes |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>model_version</th>\n",
       "      <th>time</th>\n",
       "      <th>raw_data</th>\n",
       "      <th>raw_prediction</th>\n",
       "      <th>groundtruth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:21+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"House Passes Sweeping...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.524790823459...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:26+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"iOS 14 lets deaf user...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.484371215105...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:09+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"YouTube introduces Vi...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.512727320194...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:12+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"A Filmmaker Put Away ...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.429436147212...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:23+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"Scrollability\"}]}</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.510410726070...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:15+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"A native Mac app wrap...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.482146084308...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:17+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"Astronauts Dock With ...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.516605079174...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    model model_version                      time  \\\n",
       "0  txtcls        swivel 2020-06-26 03:15:21+00:00   \n",
       "1  txtcls        swivel 2020-06-26 03:15:26+00:00   \n",
       "2  txtcls        swivel 2020-06-26 03:15:09+00:00   \n",
       "3  txtcls        swivel 2020-06-26 03:15:12+00:00   \n",
       "4  txtcls        swivel 2020-06-26 03:15:23+00:00   \n",
       "5  txtcls        swivel 2020-06-26 03:15:15+00:00   \n",
       "6  txtcls        swivel 2020-06-26 03:15:17+00:00   \n",
       "\n",
       "                                            raw_data  \\\n",
       "0  {\"instances\": [{\"text\": \"House Passes Sweeping...   \n",
       "1  {\"instances\": [{\"text\": \"iOS 14 lets deaf user...   \n",
       "2  {\"instances\": [{\"text\": \"YouTube introduces Vi...   \n",
       "3  {\"instances\": [{\"text\": \"A Filmmaker Put Away ...   \n",
       "4         {\"instances\": [{\"text\": \"Scrollability\"}]}   \n",
       "5  {\"instances\": [{\"text\": \"A native Mac app wrap...   \n",
       "6  {\"instances\": [{\"text\": \"Astronauts Dock With ...   \n",
       "\n",
       "                                      raw_prediction groundtruth  \n",
       "0  {\"predictions\": [{\"confidence\": 0.524790823459...        None  \n",
       "1  {\"predictions\": [{\"confidence\": 0.484371215105...        None  \n",
       "2  {\"predictions\": [{\"confidence\": 0.512727320194...        None  \n",
       "3  {\"predictions\": [{\"confidence\": 0.429436147212...        None  \n",
       "4  {\"predictions\": [{\"confidence\": 0.510410726070...        None  \n",
       "5  {\"predictions\": [{\"confidence\": 0.482146084308...        None  \n",
       "6  {\"predictions\": [{\"confidence\": 0.516605079174...        None  "
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "SELECT * FROM `txtcls_eval.swivel`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Provide the ground truth for the raw prediction input\n",
    "\n",
    "Notice the groundtruth is missing. We'll update the evaluation table to contain the ground truth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "UPDATE `txtcls_eval.swivel`\n",
    "SET \n",
    "    groundtruth = '{\"predictions\": [{\"source\": \"techcrunch\"}]}'\n",
    "WHERE\n",
    "    raw_data = '{\"instances\": [{\"text\": \"YouTube introduces Video Chapters to make it easier to navigate longer videos\"}]}';"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "UPDATE `txtcls_eval.swivel`\n",
    "SET \n",
    "    groundtruth = '{\"predictions\": [{\"source\": \"nytimes\"}]}'\n",
    "WHERE\n",
    "    raw_data = '{\"instances\": [{\"text\": \"A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison\"}]}';"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "UPDATE `txtcls_eval.swivel`\n",
    "SET \n",
    "    groundtruth = '{\"predictions\": [{\"source\": \"github\"}]}'\n",
    "WHERE\n",
    "    raw_data = '{\"instances\": [{\"text\": \"A native Mac app wrapper for WhatsApp Web\"}]}';"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "UPDATE `txtcls_eval.swivel`\n",
    "SET \n",
    "    groundtruth = '{\"predictions\": [{\"source\": \"nytimes\"}]}'\n",
    "WHERE\n",
    "    raw_data = '{\"instances\": [{\"text\": \"Astronauts Dock With Space Station After Historic SpaceX Launch\"}]}';"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "UPDATE `txtcls_eval.swivel`\n",
    "SET \n",
    "    groundtruth = '{\"predictions\": [{\"source\": \"nytimes\"}]}'\n",
    "WHERE\n",
    "    raw_data = '{\"instances\": [{\"text\": \"House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force\"}]}';"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "UPDATE `txtcls_eval.swivel`\n",
    "SET \n",
    "    groundtruth = '{\"predictions\": [{\"source\": \"github\"}]}'\n",
    "WHERE\n",
    "    raw_data = '{\"instances\": [{\"text\": \"Scrollability\"}]}';"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "UPDATE `txtcls_eval.swivel`\n",
    "SET \n",
    "    groundtruth = '{\"predictions\": [{\"source\": \"techcrunch\"}]}'\n",
    "WHERE\n",
    "    raw_data = '{\"instances\": [{\"text\": \"iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks\"}]}';"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can confirm that the ground truch has been properly added to the table. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>model_version</th>\n",
       "      <th>time</th>\n",
       "      <th>raw_data</th>\n",
       "      <th>raw_prediction</th>\n",
       "      <th>groundtruth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:15+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"A native Mac app wrap...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.482146084308...</td>\n",
       "      <td>{\"predictions\": [{\"source\": \"github\"}]}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:23+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"Scrollability\"}]}</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.510410726070...</td>\n",
       "      <td>{\"predictions\": [{\"source\": \"github\"}]}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:17+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"Astronauts Dock With ...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.516605079174...</td>\n",
       "      <td>{\"predictions\": [{\"source\": \"nytimes\"}]}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:12+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"A Filmmaker Put Away ...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.429436147212...</td>\n",
       "      <td>{\"predictions\": [{\"source\": \"nytimes\"}]}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:21+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"House Passes Sweeping...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.524790823459...</td>\n",
       "      <td>{\"predictions\": [{\"source\": \"nytimes\"}]}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:09+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"YouTube introduces Vi...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.512727320194...</td>\n",
       "      <td>{\"predictions\": [{\"source\": \"techcrunch\"}]}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:26+00:00</td>\n",
       "      <td>{\"instances\": [{\"text\": \"iOS 14 lets deaf user...</td>\n",
       "      <td>{\"predictions\": [{\"confidence\": 0.484371215105...</td>\n",
       "      <td>{\"predictions\": [{\"source\": \"techcrunch\"}]}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    model model_version                      time  \\\n",
       "0  txtcls        swivel 2020-06-26 03:15:15+00:00   \n",
       "1  txtcls        swivel 2020-06-26 03:15:23+00:00   \n",
       "2  txtcls        swivel 2020-06-26 03:15:17+00:00   \n",
       "3  txtcls        swivel 2020-06-26 03:15:12+00:00   \n",
       "4  txtcls        swivel 2020-06-26 03:15:21+00:00   \n",
       "5  txtcls        swivel 2020-06-26 03:15:09+00:00   \n",
       "6  txtcls        swivel 2020-06-26 03:15:26+00:00   \n",
       "\n",
       "                                            raw_data  \\\n",
       "0  {\"instances\": [{\"text\": \"A native Mac app wrap...   \n",
       "1         {\"instances\": [{\"text\": \"Scrollability\"}]}   \n",
       "2  {\"instances\": [{\"text\": \"Astronauts Dock With ...   \n",
       "3  {\"instances\": [{\"text\": \"A Filmmaker Put Away ...   \n",
       "4  {\"instances\": [{\"text\": \"House Passes Sweeping...   \n",
       "5  {\"instances\": [{\"text\": \"YouTube introduces Vi...   \n",
       "6  {\"instances\": [{\"text\": \"iOS 14 lets deaf user...   \n",
       "\n",
       "                                      raw_prediction  \\\n",
       "0  {\"predictions\": [{\"confidence\": 0.482146084308...   \n",
       "1  {\"predictions\": [{\"confidence\": 0.510410726070...   \n",
       "2  {\"predictions\": [{\"confidence\": 0.516605079174...   \n",
       "3  {\"predictions\": [{\"confidence\": 0.429436147212...   \n",
       "4  {\"predictions\": [{\"confidence\": 0.524790823459...   \n",
       "5  {\"predictions\": [{\"confidence\": 0.512727320194...   \n",
       "6  {\"predictions\": [{\"confidence\": 0.484371215105...   \n",
       "\n",
       "                                   groundtruth  \n",
       "0      {\"predictions\": [{\"source\": \"github\"}]}  \n",
       "1      {\"predictions\": [{\"source\": \"github\"}]}  \n",
       "2     {\"predictions\": [{\"source\": \"nytimes\"}]}  \n",
       "3     {\"predictions\": [{\"source\": \"nytimes\"}]}  \n",
       "4     {\"predictions\": [{\"source\": \"nytimes\"}]}  \n",
       "5  {\"predictions\": [{\"source\": \"techcrunch\"}]}  \n",
       "6  {\"predictions\": [{\"source\": \"techcrunch\"}]}  "
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "SELECT * FROM `txtcls_eval.swivel`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute evaluation metrics\n",
    "\n",
    "With the raw prediction input, the model output and the groundtruth in one place, we can evaluation how our model performs. And how the model performs across various aspects (e.g. over time, different model versions, different labels, etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import precision_recall_fscore_support as score\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using regex we can extract the model predictions, to have an easier to read format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>model_version</th>\n",
       "      <th>time</th>\n",
       "      <th>text</th>\n",
       "      <th>prediction</th>\n",
       "      <th>confidence</th>\n",
       "      <th>groundtruth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:12+00:00</td>\n",
       "      <td>A Filmmaker Put Away for Tax Fraud Takes Us In...</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.42</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:21+00:00</td>\n",
       "      <td>House Passes Sweeping Policing Bill Targeting ...</td>\n",
       "      <td>nytimes</td>\n",
       "      <td>0.52</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:23+00:00</td>\n",
       "      <td>Scrollability</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.51</td>\n",
       "      <td>github</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:09+00:00</td>\n",
       "      <td>YouTube introduces Video Chapters to make it e...</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.51</td>\n",
       "      <td>techcrunch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:26+00:00</td>\n",
       "      <td>iOS 14 lets deaf users set alerts for importan...</td>\n",
       "      <td>nytimes</td>\n",
       "      <td>0.48</td>\n",
       "      <td>techcrunch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:17+00:00</td>\n",
       "      <td>Astronauts Dock With Space Station After Histo...</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.51</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:15+00:00</td>\n",
       "      <td>A native Mac app wrapper for WhatsApp Web</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.48</td>\n",
       "      <td>github</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    model model_version                      time  \\\n",
       "0  txtcls        swivel 2020-06-26 03:15:12+00:00   \n",
       "1  txtcls        swivel 2020-06-26 03:15:21+00:00   \n",
       "2  txtcls        swivel 2020-06-26 03:15:23+00:00   \n",
       "3  txtcls        swivel 2020-06-26 03:15:09+00:00   \n",
       "4  txtcls        swivel 2020-06-26 03:15:26+00:00   \n",
       "5  txtcls        swivel 2020-06-26 03:15:17+00:00   \n",
       "6  txtcls        swivel 2020-06-26 03:15:15+00:00   \n",
       "\n",
       "                                                text  prediction confidence  \\\n",
       "0  A Filmmaker Put Away for Tax Fraud Takes Us In...  techcrunch       0.42   \n",
       "1  House Passes Sweeping Policing Bill Targeting ...     nytimes       0.52   \n",
       "2                                      Scrollability  techcrunch       0.51   \n",
       "3  YouTube introduces Video Chapters to make it e...  techcrunch       0.51   \n",
       "4  iOS 14 lets deaf users set alerts for importan...     nytimes       0.48   \n",
       "5  Astronauts Dock With Space Station After Histo...  techcrunch       0.51   \n",
       "6          A native Mac app wrapper for WhatsApp Web  techcrunch       0.48   \n",
       "\n",
       "  groundtruth  \n",
       "0     nytimes  \n",
       "1     nytimes  \n",
       "2      github  \n",
       "3  techcrunch  \n",
       "4  techcrunch  \n",
       "5     nytimes  \n",
       "6      github  "
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project $PROJECT\n",
    "SELECT\n",
    "  model,\n",
    "  model_version,\n",
    "  time,\n",
    "  REGEXP_EXTRACT(raw_data, r'.*\"text\": \"(.*)\"') AS text,\n",
    "  REGEXP_EXTRACT(raw_prediction, r'.*\"source\": \"(.*?)\"') AS prediction,\n",
    "  REGEXP_EXTRACT(raw_prediction, r'.*\"confidence\": (0.\\d{2}).*') AS confidence,\n",
    "  REGEXP_EXTRACT(groundtruth, r'.*\"source\": \"(.*?)\"') AS groundtruth,\n",
    "FROM\n",
    "  `txtcls_eval.swivel`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = '''\n",
    "SELECT\n",
    "  model,\n",
    "  model_version,\n",
    "  time,\n",
    "  REGEXP_EXTRACT(raw_data, r'.*\"text\": \"(.*)\"') AS text,\n",
    "  REGEXP_EXTRACT(raw_prediction, r'.*\"source\": \"(.*?)\"') AS prediction,\n",
    "  REGEXP_EXTRACT(raw_prediction, r'.*\"confidence\": (0.\\d{2}).*') AS confidence,\n",
    "  REGEXP_EXTRACT(groundtruth, r'.*\"source\": \"(.*?)\"') AS groundtruth,\n",
    "FROM\n",
    "  `txtcls_eval.swivel`\n",
    "'''\n",
    "\n",
    "client = bigquery.Client()\n",
    "df_results = client.query(query).to_dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>model_version</th>\n",
       "      <th>time</th>\n",
       "      <th>text</th>\n",
       "      <th>prediction</th>\n",
       "      <th>confidence</th>\n",
       "      <th>groundtruth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:12+00:00</td>\n",
       "      <td>A Filmmaker Put Away for Tax Fraud Takes Us In...</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.42</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:21+00:00</td>\n",
       "      <td>House Passes Sweeping Policing Bill Targeting ...</td>\n",
       "      <td>nytimes</td>\n",
       "      <td>0.52</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:15+00:00</td>\n",
       "      <td>A native Mac app wrapper for WhatsApp Web</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.48</td>\n",
       "      <td>github</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:23+00:00</td>\n",
       "      <td>Scrollability</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.51</td>\n",
       "      <td>github</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:09+00:00</td>\n",
       "      <td>YouTube introduces Video Chapters to make it e...</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.51</td>\n",
       "      <td>techcrunch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:17+00:00</td>\n",
       "      <td>Astronauts Dock With Space Station After Histo...</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.51</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:26+00:00</td>\n",
       "      <td>iOS 14 lets deaf users set alerts for importan...</td>\n",
       "      <td>nytimes</td>\n",
       "      <td>0.48</td>\n",
       "      <td>techcrunch</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    model model_version                      time  \\\n",
       "0  txtcls        swivel 2020-06-26 03:15:12+00:00   \n",
       "1  txtcls        swivel 2020-06-26 03:15:21+00:00   \n",
       "2  txtcls        swivel 2020-06-26 03:15:15+00:00   \n",
       "3  txtcls        swivel 2020-06-26 03:15:23+00:00   \n",
       "4  txtcls        swivel 2020-06-26 03:15:09+00:00   \n",
       "5  txtcls        swivel 2020-06-26 03:15:17+00:00   \n",
       "6  txtcls        swivel 2020-06-26 03:15:26+00:00   \n",
       "\n",
       "                                                text  prediction confidence  \\\n",
       "0  A Filmmaker Put Away for Tax Fraud Takes Us In...  techcrunch       0.42   \n",
       "1  House Passes Sweeping Policing Bill Targeting ...     nytimes       0.52   \n",
       "2          A native Mac app wrapper for WhatsApp Web  techcrunch       0.48   \n",
       "3                                      Scrollability  techcrunch       0.51   \n",
       "4  YouTube introduces Video Chapters to make it e...  techcrunch       0.51   \n",
       "5  Astronauts Dock With Space Station After Histo...  techcrunch       0.51   \n",
       "6  iOS 14 lets deaf users set alerts for importan...     nytimes       0.48   \n",
       "\n",
       "  groundtruth  \n",
       "0     nytimes  \n",
       "1     nytimes  \n",
       "2      github  \n",
       "3      github  \n",
       "4  techcrunch  \n",
       "5     nytimes  \n",
       "6  techcrunch  "
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = list(df_results.prediction)\n",
    "groundtruth = list(df_results.groundtruth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "precision, recall, fscore, support = score(groundtruth, prediction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| source     |   precision |   recall |   fscore |   support |\n",
      "|------------+-------------+----------+----------+-----------|\n",
      "| github     |         0   | 0        | 0        |         2 |\n",
      "| nytimes    |         0.5 | 0.333333 | 0.4      |         3 |\n",
      "| techcrunch |         0.2 | 0.5      | 0.285714 |         2 |\n"
     ]
    }
   ],
   "source": [
    "from tabulate import tabulate\n",
    "sources = list(CLASSES.keys())\n",
    "results = list(zip(sources, precision, recall, fscore, support))\n",
    "print(tabulate(results, headers = ['source', 'precision', 'recall', 'fscore', 'support'],\n",
    "         tablefmt='orgtbl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or a full classification report from the sklearn library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "      github       0.00      0.00      0.00         2\n",
      "     nytimes       0.50      0.33      0.40         3\n",
      "  techcrunch       0.20      0.50      0.29         2\n",
      "\n",
      "    accuracy                           0.29         7\n",
      "   macro avg       0.23      0.28      0.23         7\n",
      "weighted avg       0.27      0.29      0.25         7\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_true=groundtruth, y_pred=prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can also examine a confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAEWCAYAAAB7QRxFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de5xVVf3/8dcbEK94l0EFb0mWmuINv181b9/yh6Z5q7x2MZW0qG/fyrTsq2VRVuY3TYxQ0cxEK8UoUTTNG2qKBoKKiog5ouAlEfAKfH5/7D26GWbm7DNz9pyzh/eTx3nM2XuvvdaaPcNn1ll77bUUEZiZWXn1qncFzMysaxzIzcxKzoHczKzkHMjNzErOgdzMrOQcyM3MSs6B3LpM0uqS/iJpgaQ/diGf4yTdUsu61YOkmyR9vt71sJWHA/lKRNKxkqZIWiTphTTg7FWDrD8FNAEbRMSnO5tJRPw+Ig6oQX2WI2lfSSHp+lb7d0z335Ezn+9LuqpSuog4MCJ+28nqmlXNgXwlIekbwC+BH5ME3c2Ai4FDa5D95sCTEbGkBnkV5SVgD0kbZPZ9HniyVgUo4f9T1u38S7cSkLQOcA7wlYi4PiIWR8S7EfGXiDgtTbOqpF9Kmpu+filp1fTYvpKaJX1T0vy0NX9CeuwHwFnAUWlL/8TWLVdJW6Qt3z7p9hckzZa0UNIzko7L7L8nc94ekh5Mu2welLRH5tgdkn4oaXKazy2SNuzgMrwD3AAcnZ7fG/gM8PtW1+oCSc9Jel3SQ5I+mu4fBnw3831Oy9RjpKTJwBvAVum+k9Ljv5b0p0z+P5V0myTl/gGaVeBAvnL4T2A1YHwHac4E/gMYAuwIDAW+lzk+AFgH2BQ4ERglab2IOJuklX9tRKwVEZd1VBFJawIXAgdGRD9gD2BqG+nWB25M024AnA/c2KpFfSxwAtAf6At8q6OygSuBz6Xv/x/wKDC3VZoHSa7B+sDVwB8lrRYRN7f6PnfMnPNZYDjQD3i2VX7fBHZI/0h9lOTafT48N4bVkAP5ymED4OUKXR/HAedExPyIeAn4AUmAavFuevzdiJgILAK26WR9lgHbS1o9Il6IiEfbSPMJ4KmI+F1ELImIccBM4JBMmssj4smIeBP4A0kAbldE3AusL2kbkoB+ZRtproqIV9IyfwGsSuXv84qIeDQ9591W+b0BHE/yh+gq4KsR0VwhP7OqOJCvHF4BNmzp2mjHJizfmnw23fdeHq3+ELwBrFVtRSJiMXAUcArwgqQbJX0oR31a6rRpZvvFTtTnd8AIYD/a+ISSdh89nnbnvEbyKaSjLhuA5zo6GBEPALMBkfzBMaspB/KVw33AW8BhHaSZS3LTssVmrNjtkNdiYI3M9oDswYiYFBEfBzYmaWVfkqM+LXV6vpN1avE74MvAxLS1/J606+N0kr7z9SJiXWABSQAGaK87pMNuEklfIWnZzwW+3fmqm7XNgXwlEBELSG5IjpJ0mKQ1JK0i6UBJP0uTjQO+J2mj9KbhWSRdAZ0xFdhb0mbpjdbvtByQ1CTpk2lf+dskXTRL28hjIvDBdMhkH0lHAdsCf+1knQCIiGeAfUjuCbTWD1hCMsKlj6SzgLUzx+cBW1QzMkXSB4EfkXSvfBb4tqQOu4DMquVAvpKIiPOBb5DcwHyJpDtgBMlIDkiCzRTgEWA68HC6rzNl3Qpcm+b1EMsH314kNwDnAq+SBNUvt5HHK8DBadpXSFqyB0fEy52pU6u874mItj5tTAJuIhmS+CzJp5hst0nLw06vSHq4UjlpV9ZVwE8jYlpEPEUy8uV3LSOCzGpBvnluZlZubpGbmZWcA7mZWQ1JGiTp7+nop0cl/XcbaSTpQkmzJD0iaefMsWGSnkiPnZGnTAdyM7PaWgJ8MyI+TPKQ3VckbdsqzYHA4PQ1HPg1vPfE8aj0+LbAMW2cuwIHcjOzGkofcns4fb8QeJzln3+AZI6jKyNxP7CupI1JnqieFRGzI+Id4BpyzIfU0QMidfXWko7H5pqVwXq7jah3FVYKb/7zoi7PXbP6TiNyx5y3po76EklLusWYiBjTOp2kLYCdgH+0OrQpy4+Iak73tbV/90r1adhAbmbWqNKgvULgzpK0FnAd8PWIeL314bay7WB/hxzIzcwAajgDsaRVSIL47yPi+jaSNAODMtsDSZ6t6NvO/g65j9zMDKBX7/yvDqRTFF8GPJ4+iNeWCcDn0tEr/wEsiIgXSGbfHCxpS0l9SaZdnlCp6m6Rm5kB1G6K+D1JpmOYLqlliubvkswVRESMJpmC4iBgFsmEbyekx5ZIGkHylHFvYGw7s4Mux4HczAxq1rUSEffQdl93Nk0AX2nn2ESSQJ+bA7mZGdSyRd7tHMjNzKCmNzu7mwO5mRm4RW5mVnoVRqM0MgdyMzNw14qZWem5a8XMrOTcIjczKzkHcjOzkuvtm51mZuXmPnIzs5Jz14qZWcm5RW5mVnJukZuZlZxb5GZmJedH9M3MSs5dK2ZmJeeuFTOzknOL3Mys5GoYyCWNBQ4G5kfE9m0cPw04Lt3sA3wY2CgiXpU0B1gILAWWRMSulcor758gM7Na6tU7/6uyK4Bh7R2MiJ9HxJCIGAJ8B7gzIl7NJNkvPV4xiINb5GZmiRr2kUfEXZK2yJn8GGBcV8pzi9zMDJKulbyvWhUprUHScr8uszuAWyQ9JGl4nnzcIjczg6pa5GmAzQbZMRExphOlHgJMbtWtsmdEzJXUH7hV0syIuKujTBzIzcwAVRHI06DdmcDd2tG06laJiLnp1/mSxgNDgQ4DubtWzMxIAnneV43KWwfYB/hzZt+akvq1vAcOAGZUysstcjMzQL1qd7NT0jhgX2BDSc3A2cAqABExOk12OHBLRCzOnNoEjE//WPQBro6ImyuV50BeA5PvvoufnjuSZUuXcfiRn+bEk3Pdn7Aq+ToXa2DTulz6w8/RtMHaLItg7HWTGTXujnpXq9vUqqUNEBHH5EhzBckwxey+2cCO1ZbnQN5FS5cu5ccjz+E3l1xOU1MTxx71Kfbdb38+sPXW9a5aj+LrXLwlS5dxxvnXM3VmM2utsSr3Xn06t/1jJjNnv1jvqnWLWgby7uY+8i6aMf0RBg3anIGDBrFK374MO+gT3PH32+pdrR7H17l4L778OlNnNgOw6I23mfnMi2yy0bp1rlX36e4+8lpyIO+i+fPmMWDjAe9t929qYt68eXWsUc/k69y9Ntt4fYZsM5AHZ8ypd1W6j6p4NZjCu1YkHQHsRTLI/Z6IGF90md0piBX2NeJf7LLzde4+a67el3HnncRp513HwsVv1bs63abMv0+FtsglXQycAkwnGULzJUmjOkg/XNIUSVMuu6QWQzSL19Q0gBdfeL8Pcf68efTv37+ONeqZfJ27R58+vRh33slce9MU/nz7tHpXp1v16tUr96vRFN0i3wfYPiICQNJvSYJ6m7KD7N9a0kYTrAFtt/1H+Ne/5tDc/BxN/Zu4eeKN/OTnv6h3tXocX+fuMfrs43jimRe58Krb612VblfmFnnRgfwJYDPg2XR7EPBIwWV2qz59+vCdM8/i1OEnsWzZUg47/Ei23npwvavV4/g6F2+PIVtx3MG7M/3J57n/mjMAOPuiCUy657E616yblDeOo7SxXNtMpb+Q9ImvA+wGPJBu7w7cGxEfq5RHWVrkZh1Zb7cR9a7CSuHNf17U5TC84ReuyR1zXr7i6IYK+0W1yM8rKF8zs0K4a6WViLiziHzNzIpSy0f0u1uhfeSSFsJ7XSR9SeYaWBwRaxdZrplZtdwib0dE9MtuSzqMZEpGM7OGUuZA3q0DIiPiBmD/7izTzCyPMj+iX3TXyhGZzV7AruDRKGbWeBoxQOdV9DjyQzLvlwBzgEMLLtPMrHrljeOF95GfUGT+Zma10oiP3udVdNfKRsDJwBbZsiLii0WWa2ZWLXettO/PwN3A34ClBZdlZtZ55Y3jhQfyNSLi9ILLMDPrsjK3yIvuFPqrpIMKLsPMrMtqOfxQ0lhJ8yXNaOf4vpIWSJqavs7KHBsm6QlJsySdkafuhbTIM090CviupLeBd9Pt8JOdZtZoatwivwK4CLiygzR3R8TBrerQGxgFfBxoBh6UNCEiOpyCsqi5VvpVTmVm1jhqOddKRNwlaYtOnDoUmBURswEkXUMyZLvDQF70CkErrI7b1j4zs3qrpmslu5pZ+hreiSL/U9I0STdJ2i7dtynwXCZNc7qvQ0V1rawGrAlsKGk93r8fvDawSRFlmpl1RTVdK9nVzDrpYWDziFiU3ke8ARhM22NnKj4NX9SolS8BXycJ2g9n9r9O0v9jZtZQunPQSkS8nnk/UdLFkjYkaYEPyiQdCMytlF9RfeQXABdI+mpE/KqIMszMaqk7hx9KGgDMi4iQNJSkm/sV4DVgsKQtgeeBo4FjK+VXVNfK/hFxO/B8q4mzAIiI64so18yss3rV8GanpHHAviTdy83A2STrMRARo4FPAadKWgK8CRydLlK/RNIIYBLQGxgbEY9WKq+orpV9gNtJJs1qGYaY/epAbmYNpZYN8og4psLxi0iGJ7Z1bCIwsZryiupaOTt9O4P3Azjp+wWShkTE1CLKNjPrjFq2yLtb0U927gKcAmxMcuNzOMnHjUskfbvgss3McpPyvxpN0XOtbADsHBGLACSdDfwJ2Bt4CPhZweWbmeVS5rlWig7kmwHvZLbfJRk7+Wb62L6ZWUMocRwvPJBfDdwv6c/p9iHAOElrUuGRUzOz7uSFJdoRET+UNBHYi+SG5ykRMSU9fFyRZZuZVcMt8g5ExEMk/eFmZg3LfeRmZiVX4jjuQG5mBm6Rm5mVXonjuAO5mRmU+8lOB/KV2Df/8ni9q2DWMNy1YmZWciWO4w7kZmbgFrmZWemVOI47kJuZgW92mpmVnrtWzMxKrsyBvLzTfZmZ1VAtF5aQNFbSfEkz2jl+nKRH0te9knbMHJsjabqkqZKmtHV+a26Rm5lR8xb5FSRrcl7ZzvFngH0i4t+SDgTGALtnju8XES/nLcyB3MyMmi++fJekLTo4fm9m835gYFfKc9eKmRnJqJW8L0nDJU3JvIZ3oegTgZsy2wHcIumhvPm6RW5mBvSqokkeEWNIukO6RNJ+JIF8r8zuPSNirqT+wK2SZkbEXR3lU7FFLukISf3S92dI+oOkIV2pvJlZo6nlzc585WkH4FLg0Ih4pWV/RMxNv84HxgNDK+WVp2vl+xGxUNIeJGtuXguM7kzFzcwalaTcrxqUtRlwPfDZiHgys3/NTMN5TeAAoM2RL1l5ulaWpl8PBi6OiOskfa/qmpuZNbBaPtgpaRywL7ChpGbgbGAVgIgYDZwFbABcnP5hWBIRuwJNwPh0Xx/g6oi4uVJ5eQL5C5JGAcOAXSX1xTdJzayHqeUj+hFxTIXjJwEntbF/NrDjimd0LE9A/gxwJ/CJiPg3sCFwRrUFmZk1MlXxr9G02yKXtHZm8+bMvkXA5ILrZWbWrUo8Z1aHXSuPkoxnzH57LdsBbFZgvczMulWZ51ppN5BHxKDurIiZWT2VOI7nu2kp6WhJ303fD5S0S7HVMjPrXr2k3K9Gk+eBoIuA/YDPprvewOPIzayHqeYR/UaTZ/jhHhGxs6R/AkTEq+kQRDOzHqMBG9q55Qnk70rqRXKDE0kbAMsKrZWZWTdrxC6TvPL0kY8CrgM2kvQD4B7gp4XWysysm6mKV6Op2CKPiCslPQR8LN316Yio+Oy/mVmZ9Mjhh630Bt4l6V7x4/lm1uM04D3M3PKMWjkTGAdsQrKKxdWSvlN0xczMulNPH7VyPLBLRLwBIGkk8BDwkyIrZmbWnXp618qzrdL1AWYXUx0zs/powIZ2bh1NmvV/JH3ibwCPSpqUbh9AMnLFzKzH6Kkt8paRKY8CN2b2319cdczM6qO8YbzjSbMu686KmJnVU+8S961U7COX9AFgJLAtsFrL/oj4YIH1KpXJd9/FT88dybKlyzj8yE9z4snD612lHuf4nTdm+wFrsfDtJYy87Zl6V6dHGti0Lpf+8HM0bbA2yyIYe91kRo27o97V6jZl7lrJMyb8CuBykk8eBwJ/AK4psE6lsnTpUn488hwuHn0p4yfcyM0T/8rTs2bVu1o9zv3Pvsaoyc/Vuxo92pKlyzjj/OvZ6cgfsc/nzuNLR+3Nh7YaUO9qdRsp/6tyXhorab6kNh+eVOJCSbMkPSJp58yxYZKeSI/lWo0tTyBfIyImAUTE0xHxPZLZEA2YMf0RBg3anIGDBrFK374MO+gT3PH32+pdrR5n1itvsvjdpZUTWqe9+PLrTJ3ZDMCiN95m5jMvsslG69a5Vt2nxtPYXkGyznF7DgQGp6/hwK8BJPUmmRblQJJekGMkbVux7jkq9LaSzxxPSzpF0iFA/xznIekDklZN3+8r6WuSetRvxvx58xiw8futlv5NTcybN6+ONTLrus02Xp8h2wzkwRlz6l2VblPLFnlE3AW82kGSQ4ErI3E/sK6kjYGhwKyImB0R75D0fhxaqbw8gfx/gLWArwF7AicDX8xxHiSTbS2VtDVwGbAlcHV7iSUNlzRF0pTLLhmTs4j6imRSyOWUua/NbM3V+zLuvJM47bzrWLj4rXpXp9tIqub1XqxKX9XeGNsUyPYVNqf72tvfoTyTZv0jfbuQ9xeXyGtZRCyRdDjwy4j4Vcu85u2UNQYYA/DWkjYiZANqahrAiy+8+N72/Hnz6N8/1wcWs4bTp08vxp13MtfeNIU/3z6t3tXpVr2raIBlY1UntVVY6zWSs/s71NEDQeM7yiAijqiUOclc5scAnwcOSfetkuO80thu+4/wr3/Nobn5OZr6N3HzxBv5yc9/Ue9qmXXK6LOP44lnXuTCq26vd1W6XTePPmwGsusiDwTmAn3b2d+hjlrkF3Wmdq2cAJwCjIyIZyRtCVxVg3wbRp8+ffjOmWdx6vCTWLZsKYcdfiRbbz243tXqcU7YdRMGb7Qma/XtzY+Gbc2Nj7/Efc8uqHe1epQ9hmzFcQfvzvQnn+f+a5LBEmdfNIFJ9zxW55p1j24O5BOAEZKuAXYHFkTEC5JeAgansfJ54Gjg2EqZdfRAUJeHXkTEY5JOBzZLt58Bzu1qvo3mo3vvw0f33qfe1ejRLp9SsVFiXXTv1NmsvtOIelejbmp5b0vSOGBfYENJzcDZpL0RETEamAgcBMwimQblhPTYEkkjgEkk04ePjYhHK5WXdz7yTklHuJxH8nFhS0lDgHMi4pNFlmtmVq1atsgj4pgKxwP4SjvHJpIE+tyKXiTi+yTDaV4DiIipJCNXzMwaSi2HH3a33C1ySatGxNtV5r8kIha0+shSitEoZrZy6dOIETqnPCsEDZU0HXgq3d5R0q9y5j9D0rFAb0mD0/Pu7Xx1zcyKUeYWeZ6ulQuBg4FXACJiGvkf0f8qsB3wNslyca8DX6++mmZmxarxI/rdKk/XSq+IeLZV90iuSS/S5eHOTF9mZg2rAeNzbnkC+XOShgKRTujyVeDJPJlL2hX4LrBFtqyI2KH6qpqZFafE05HnCuSnknSvbAbMA/6W7svj98BpwHRgWWcqaGbWHXr0whIRMZ/k6aLOeCkiJnTyXDOzblPiOJ5rhaBLaGPIYETkme3rbEmXAreR3PBsOff6aippZlY0lXjVzjxdK3/LvF8NOJzlp1nsyAnAh0geTW3pWgnAgdzMGkqPbpFHxLXZbUm/A27Nmf+OEfGRzlTMzKw7lTmQd+YR/S2BzXOmvT/PMkVmZvVWzcISjSZPH/m/eb+PvBfJ8kW5FgQF9gI+L+kZkj5ykcwX4+GHZtZQehc981SBOgzk6VqdO5LMiwvJij/VzJXS0eKjZmYNoxGf2Myrw0AeESFpfETsUk2mktaOiNdJloczM2t4Ze4jzzNq5QFJO0fEw1XkezXJ/CwPseI6dAFsVUVeZmaFK3GDvMM1O/tExBKSfu6TJT0NLOb9fu6d2zs3Ig5Ov3rucTMrhV49dBz5A8DOwGGdzVzSbRHxX5X2mZnVW49skZN2h0TE09VmKmk1YA2S9erW4/2ulbWBTarNz8ysaH1q2EkuaRhwAcm6m5dGxLmtjp8GHNdSNPBhYKOIeFXSHJL7i0tJFufZtWLdOzi2kaRvtHcwIs7v4Nwvkcw7vgmQ7Vt/HRhVqVJmZt2tVi3ydJbYUcDHgWbgQUkTIuKxljQR8XPg52n6Q4D/iYhXM9nsFxEv5y2zo0DeG1gLqu84iogLgAskfTUi8q4mZGZWNzUcfjgUmBURswEkXQMcCjzWTvpjSBbe6bSOAvkLEXFOVzIneRhoKXB1RLzWxbzMzApTTRyXNBzIThw4JiLGpO83Zfn5qJqB3dvJZw2S521GZHYHcIukAH6TybddFfvIu+hokomzpkiaAlwO3FLlQ0VmZoWr5sHONLi2F2Dbip3txbxDgMmtulX2jIi5kvoDt0qaGRF3dVSfjure5ZElETErIs4EPkgytnws8C9JP5C0flfzNzOrlRqu2dkMDMpsDwTmtpP2aFp1q0TE3PTrfGA8SVdNx3Vv70CrvxCdJmkH4HySjv3rgE+R3PS8vRb5m5nVQg0D+YPAYElbSupLEqxXWGBH0jrAPsCfM/vWlNSv5T1wADCjUoF5nuzsNEkPAa8BlwKnR0TL4hL/kLRnkWWbmVWjVrc6I2KJpBHAJJJBI2Mj4lFJp6THR6dJDyfpal6cOb0JGJ/OsNiH5P7izZXKLDSQA58FdiKZ+vb0lukfI+KciDii4LLNzHKr5QNBETERmNhq3+hW21cAV7TaN5tkosKqFB3IzydpkT9MZqk3M7NG04jzjOdVdCAfGBGeytbMGl6JpyMvvO73SvJSb2bW8Gp4s7PbFd0i3wv4glcIspXVF8/6Sr2rYDm5a6V9Bxacv5lZTZS5a6XQQB4RzxaZv5lZrbhFbmZWcuUN4w7kZmYA9HaL3Mys3Eocxx3IzcwAVOLOFQdyMzPcIjczK71ebpGbmZWbW+RmZiXXiI/e5+VAbmYG9CpvHHcgNzMDj1oxMyu9EvesOJCbmUG5W+RlnvDLzKxmein/qxJJwyQ9IWmWpDPaOL6vpAWSpqavs/Ke2xa3yM3MqN2oFUm9gVHAx4Fm4EFJEyLisVZJ746Igzt57vJ1r0nNzcxKTlW8KhgKzIqI2RHxDnANcGjOanTqXAdyMzOqW+pN0nBJUzKv4ZmsNgWey2w3p/ta+09J0yTdJGm7Ks9djrtWzMyobj7yiBgDjKkiq2i1/TCweUQsknQQcAMwOOe5K3CL3MwMatm30gwMymwPBOZmE0TE6xGxKH0/EVhF0oZ5zm2LA7mZGdV1rVTwIDBY0paS+gJHAxOyCSQNULq2nKShJLH4lTzntsVdK2Zm1G6pt4hYImkEMAnoDYyNiEclnZIeHw18CjhV0hLgTeDoiAigzXMrlelAbmYGNV20M+0umdhq3+jM+4uAi/KeW4kDuZkZ5X6y04HczAzPtWJmVnoljuMO5GZmACpxk9yB3MwMd62YmZVeieO4A7mZGVDqSO5AbmaGhx+u9CbffRc/PXcky5Yu4/AjP82JJw+vfJJV5fidN2b7AWux8O0ljLztmXpXp0da2a9xmfvIPddKFy1dupQfjzyHi0dfyvgJN3LzxL/y9KxZ9a5Wj3P/s68xavJzlRNap63s11jK/2o0DuRdNGP6IwwatDkDBw1ilb59GXbQJ7jj77fVu1o9zqxX3mTxu0vrXY0ebWW/xqriX6NxIO+i+fPmMWDjAe9t929qYt68eXWskZl1Rplb5IX2kUvaFNg8W05E3FVkmd0t2pjzvcwPFpitrMr8v7awQC7pp8BRwGNAy+e1ANoN5OlyScMBLrr4N6W4adjUNIAXX3jxve358+bRv3//OtbIzDqlxJG8yBb5YcA2EfF23hOyyye9taTy8kaNYLvtP8K//jWH5ubnaOrfxM0Tb+QnP/9FvatlZlXKsWBEwyoykM8GVgFyB/Iy6tOnD9858yxOHX4Sy5Yt5bDDj2TrrQfXu1o9zgm7bsLgjdZkrb69+dGwrbnx8Ze479kF9a5Wj7KyX+PyhnFQsihFDTOUfkXShbIpsCNwG5lgHhFfy5NPWVrkZfbNvzxe7yqY1cSowz/c5Tj85Lw3csecDzat0VBxv4gW+ZT060PkWGvOzKwRNOKwwrxqHsgj4rcAktYE3oqIpel2b2DVWpdnZlYLtewilzQMuIBk3c1LI+LcVsePA05PNxcBp0bEtPTYHGAhySCRJRGxa6XyihxHfhuwemZ7deBvBZZnZtZpquLVYT5Jo3UUcCCwLXCMpG1bJXsG2CcidgB+SDrII2O/iBiSJ4hDsTc7V4uIRS0bEbFI0hoFlmdm1mk1fP5jKDArIman+V4DHEoyFBuAiLg3k/5+YGBXCiyyRb5Y0s4tG5J2Ad4ssDwzs06r4ZOdmwLZSWua033tORG4KbMdwC2SHkqframoyBb514E/Spqbbm9M8oCQmVnDqaY9nn14MTUmfQ6mvazaHBEjaT+SQL5XZveeETFXUn/gVkkzKz0RX1ggj4gHJX0I2IbkG5sZEe8WVZ6ZWZdUEcmzDy+2oRkYlNkeCMxtnUjSDsClwIER8Uom77np1/mSxpN01dQnkKd2A7ZIy9lJEhFxZcFlmplVrYbDDx8EBkvaEngeOBo4drmypM2A64HPRsSTmf1rAr0iYmH6/gDgnEoFFjnXyu+ADwBTWX6uFQdyM2s4tbrXGRFLJI0AJpEMPxwbEY9KOiU9Pho4C9gAuDi9ydoyzLAJGJ/u6wNcHRE3VyqzyBb5rsC2UetHR83MCtCrhuPII2IiMLHVvtGZ9ycBJ7Vx3mySJ+KrUuSolRnAgIqpzMwaQq1Gkne/IlvkGwKPSXqA5eda+WSBZZqZdUqJJz8sNJB/v8C8zcxqqsRxvNDhh3cWlbeZWa25Rd4GSQt5fxB8X5K5yRdHxNpFlWlm1lllXqKxyBZ5v+y2pMNIBrabmTWc8obxYketLCcibgD2767yzMyqUcO5VrpdkV0rR2Q2e5GMK/eYcjNrSF5Yom2HZN4vAeaQTOVoZtZ4yhvHiwnk6cTqj0TE/xWRv5lZrQVhJ4wAAAs0SURBVJU4jhfTR54u7+YHf8ysNHpJuV+NpsiulXslXQRcCyxu2RkRDxdYpplZpzRgfM6tyEC+R/o1OwVj4JErZmY1VeQ48v2KytvMrNbK3CIvbBy5pB9LWjezvZ6kHxVVnplZV6iKf42myAeCDoyI11o2IuLfwEEFlmdm1ml+IKhtvSWtGhFvA0haHVi1wPLMzDqtEQN0XkUG8quA2yRdTnKT84vAbwssz8ys0xqxyySvIm92/kzSdOC/SMba/zAiJhVVnplZV5S5RV7opFkRcVNEfCsivukgbmaNrJYLvUkaJukJSbMkndHGcUm6MD3+iKSd857bliJHrRwh6SlJCyS9LmmhpNeLKs/MrEtqFMnTKUpGAQcC2wLHSNq2VbIDgcHpazjw6yrOXUGRfeQ/Aw6JiMcLLMPMrCZq+Oj9UGBWRMwGkHQNyYSBj2XSHApcGREB3C9pXUkbA1vkOHcFRQbyeV0J4qv1Kd+dB0nDI2JMveuR16jDP1zvKlStbNe4jFbWa1xNzJE0nKQl3WJM5pptCjyXOdYM7N4qi7bSbJrz3BXUPJBn5iGfIula4Abg7ZbjEXF9rctsIMOBle4/QDfzNS6er3EFadBu7xq19Qeh9VoM7aXJc+4KimiRZ+chfwM4ILMdQE8O5GZmzcCgzPZAYG7ONH1znLuCmgfyiDih1nmamZXIg8BgSVsCzwNHA8e2SjMBGJH2ge8OLIiIFyS9lOPcFRQ5auW3bcy1Mrao8hqEP44Wz9e4eL7GXRARS4ARwCTgceAPEfGopFMknZImmwjMBmYBlwBf7ujcSmUquWlae5L+GRE7VdpnZmZdU+QDQb0krdeyIWl9ih0lY2a2UioysP6CZJWgP5Hc5PwMMLLA8szMVkqFtcgj4krgSGAe8BJwRET8rqjyiiTpHEkfS99/XdIamWOLqsxrX0l/rXUde5I2rvHE7P2WlVH6wMiXO3nuFZI+Ves6daIeX0iXf7QaK3SuFWB9YHFE/Ap4Kb0TWzoRcVZE/C3d/DqwRkfprcuWu8YRcVB2bvuV1LqkN8S6kyR3h5ZAkaNWzgZOB76T7lqFZGrbhibpfyXNlHSrpHGSvtXSopH0NWAT4O+S/p45Z6SkaZLul9SU7luuFdSq5b62pPGSHpM0WlLRf1DrStIWkh6XdImkRyXdImk7SQ9n0gyW9FBb11jSHEkbpvnMlHSppBmSfi/pY5Imp/P6DE3TrylprKQHJf1T0qHp/u0kPSBpajpR0eB6XI9OOhf4QFr3n0s6Lf3+HpH0g5ZEkj6X7psmKfsJeG9J90qa3er38tuSpqfpz0333aFkha87gf9u73c5/XR5h6Q/pT+X30vJc+6SdkvLm5Ze837p6ZtIujn9ef2suMu1komIQl7AVJKnlP6Z2fdIUeXVqM67pvVeHegHPAV8C7gC+FSaZg6wYeacIJlTBpL5Zb6Xvn/vnHR7Ufp1X+AtYCugN3BrNl1PfJHMH7EEGJJu/wE4Hvh7Zt+Pga+2c43nABtm8vkISSPkIWBs+nt2KHBDJq/j0/frAk8CawK/Ao5L9/cFVq/3tanyGs5I3x9AMkRQ6XX4K7A3sB3wRMu1A9bP/C7+MU27LclcHpBMzHQvsEar9HcAF2fK7uh3eQHJQyu9gPuAvdJrOxvYLU23Nsn9uC+k+9cBVgOeBQbV+9r2hFeRLcF3IvkpBiStpALLqpW9gD9HxJsRsRD4S45z3iH5jwRJYNkixzkPRMTsiFgKjEvL7emeiYip6fuW63QpcIKSGd+OAq7Omc/0iFgGPArclv6eTef9a38AcIakqSRBaTVgM5JA811JpwObR8SbtfjG6uCA9PVP4GHgQySz6O0P/CkiXgaIiFcz59wQEcsi4jGgKd33MeDyiHijjfTX5qzLAxHRnP48ppL8DLYBXoiIB9N8X49kfDQkP68FEfEWyURQm1fxfVs7igzkf5D0G2BdSScDfyMZ+N7IOjNR17tpIAFYyvsjgZaQXt/042bfzDmtB+8XM5i/sbyded9yna4jaRUeDDwUEa9Umc+yzPYy3r/2Ao6MiCHpa7OIeDwirgY+CbwJTJK0f+e/nboS8JPM97d1RFyW7m/vdyl73ZT52l76xZn3Hf0ut/VzzVuP7P8X64IiA/lGwJ9I/rNuA5xF8hGskd0DHCJpNUlrAZ9oI81Ckm6XSuYAu6TvDyW5R9BiqKQt077xo9JyVzppq2wSyVzMl2cO5b3G7ZkEfDXTX7tT+nUrYHZEXEjyiPQOXSiju2WvySTgi+nvKJI2ldQfuA34jKQN0v3rV8jzljSfNSqkn0P7v8ttmUnSF75bmm8/3zQtVpGB/OMRcWtEnBbJKkG3krS+Glb6UXACMI1kcq8pJH2AWWOAm7I3O9txCbCPpAdI5lLItnDuI7l5NQN4Bhjf9dqX1u9JWm+3ZPblvcbt+SFJsHlE0ox0G5I/mjPSLpcPAVd2Mv9ul35amZx+Px8n6Ya6T8lyin8C+kXyKPdI4E5J04DzK+R5M8nv+5T0mnyrnaQd/S63le87JNf6V2k9biXp3rKC1PwRfUmnkgyT2gp4OnOoHzA5Io6vaYE1JmmtiFiUtlLuAoZHxMOVzrPOkfQtYJ2I+N9618WsrIoI5OsA6wE/AbLrzS1sdTOlIUm6muTO/mrAbyPiJ3WuUo8laTzwAWD/lht0Zla9wibNMjOz7tGjH0QxM1sZOJCbmZWcA7mZWck5kNsKJC1N5/SYIemPysxE2Im83pvtUdInJZ3RQdpOzfAn6fvp6Jdc+1ulqWpmwHS+lxnV1tGsSA7k1pY30ycGtyeZguCU7EElqv7diYgJEXFuB0nqMsOfWdk5kFsldwNb6/0ZDC8mmd9jkKQDJN0n6eG05d7ypOGwdDa8e4AjWjJSZj5qSU1KZoCclr72oNUMf2m69mb5O1PSE5L+RvLkcIcknZzmM03Sda0+ZXxM0t2SnpR0cJq+t5JZBlvK/lIbeZZ5NkXrQRzIrV3pY9UHkkxIBUnAvDKSdVcXA98DPhYRO5M8BfsNSauRPAl4CPBRYEA72V8I3BkROwI7k0yAdQbwdPpp4DRJB5BMBjUUGALsImlvSbuQrC6+E8kfit1yfDvXR8RuaXmPAydmjm0B7EMyJcPo9Hs4kWRl893S/E/WivPpnwJcEBFDSGbObM5RD7Oa8/wH1pbV00e2IWmRX0YyR/izEXF/uv8/SB6cmpxOadKXZOqBD5HMUPgUgKSrgOFtlLE/8DmAdBbIBcqs8ZrKzvIHsBZJYO8HjG+ZtU/ShBzf0/aSfkTSfbMWyXwlLf6Qzt73lKTZ6fdwALBDpv98nbTsJzPn3QecKWkgyR+Kp3LUw6zmHMitLW+mrcz3pME6O8eGgFsj4phW6YZQu9kcW2b5+02rMr7eiTKuAA6LiGmSvkAyl3aLtmajFMn86NmAj6Qt3ksUcbWkf5C05CdJOikibq+yXmZd5q4V66z7gT0lbQ0gaQ1JHySZ+W5LSR9I0x3Tzvm3Aaem5/aWtDYrznrY3ix/dwGHS1pdycozh+Sobz/gBUmrAMe1OvZpSb3SOm9FsjjDJODUND2SPqhWc+qr3LMpWg/iFrl1SkS8lLZsx0laNd39vYh4UtJw4EZJL5NM0bt9G1n8NzBG0okk81KfGhH3KVm2bQZwU9pP/mGSWf4AFpGs/POwpGtJFjJ4lqT7p5L/Bf6Rpp/O8n8wngDuJFlw4ZSIeEvSpSR95w8rKfwl4LBWeR4FHC/pXeBF4Jwc9TCrOc+1YmZWcu5aMTMrOQdyM7OScyA3Mys5B3Izs5JzIDczKzkHcjOzknMgNzMruf8PW8aPFa8p100AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cm = confusion_matrix(groundtruth, prediction, labels=sources)\n",
    "\n",
    "ax= plt.subplot()\n",
    "sns.heatmap(cm, annot=True, ax = ax, cmap=\"Blues\")\n",
    "\n",
    "# labels, title and ticks\n",
    "ax.set_xlabel('Predicted labels')\n",
    "ax.set_ylabel('True labels')\n",
    "ax.set_title('Confusion Matrix') \n",
    "ax.xaxis.set_ticklabels(sources)\n",
    "ax.yaxis.set_ticklabels(sources)\n",
    "plt.savefig(\"./txtcls_cm.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examine eval metrics by model version or timestamp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By specifying the same evaluation table, two different model versions can be evaluated. Also, since the timestamp is captured, it is straightforward to evaluation model performance over time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "now = pd.Timestamp.now(tz='UTC')\n",
    "one_week_ago = now - pd.DateOffset(weeks=1)\n",
    "one_month_ago = now - pd.DateOffset(months=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_prev_week = df_results[df_results.time > one_week_ago]\n",
    "df_prev_month = df_results[df_results.time > one_month_ago]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>model_version</th>\n",
       "      <th>time</th>\n",
       "      <th>text</th>\n",
       "      <th>prediction</th>\n",
       "      <th>confidence</th>\n",
       "      <th>groundtruth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:12+00:00</td>\n",
       "      <td>A Filmmaker Put Away for Tax Fraud Takes Us In...</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.42</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:21+00:00</td>\n",
       "      <td>House Passes Sweeping Policing Bill Targeting ...</td>\n",
       "      <td>nytimes</td>\n",
       "      <td>0.52</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:15+00:00</td>\n",
       "      <td>A native Mac app wrapper for WhatsApp Web</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.48</td>\n",
       "      <td>github</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:23+00:00</td>\n",
       "      <td>Scrollability</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.51</td>\n",
       "      <td>github</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:09+00:00</td>\n",
       "      <td>YouTube introduces Video Chapters to make it e...</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.51</td>\n",
       "      <td>techcrunch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:17+00:00</td>\n",
       "      <td>Astronauts Dock With Space Station After Histo...</td>\n",
       "      <td>techcrunch</td>\n",
       "      <td>0.51</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>txtcls</td>\n",
       "      <td>swivel</td>\n",
       "      <td>2020-06-26 03:15:26+00:00</td>\n",
       "      <td>iOS 14 lets deaf users set alerts for importan...</td>\n",
       "      <td>nytimes</td>\n",
       "      <td>0.48</td>\n",
       "      <td>techcrunch</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    model model_version                      time  \\\n",
       "0  txtcls        swivel 2020-06-26 03:15:12+00:00   \n",
       "1  txtcls        swivel 2020-06-26 03:15:21+00:00   \n",
       "2  txtcls        swivel 2020-06-26 03:15:15+00:00   \n",
       "3  txtcls        swivel 2020-06-26 03:15:23+00:00   \n",
       "4  txtcls        swivel 2020-06-26 03:15:09+00:00   \n",
       "5  txtcls        swivel 2020-06-26 03:15:17+00:00   \n",
       "6  txtcls        swivel 2020-06-26 03:15:26+00:00   \n",
       "\n",
       "                                                text  prediction confidence  \\\n",
       "0  A Filmmaker Put Away for Tax Fraud Takes Us In...  techcrunch       0.42   \n",
       "1  House Passes Sweeping Policing Bill Targeting ...     nytimes       0.52   \n",
       "2          A native Mac app wrapper for WhatsApp Web  techcrunch       0.48   \n",
       "3                                      Scrollability  techcrunch       0.51   \n",
       "4  YouTube introduces Video Chapters to make it e...  techcrunch       0.51   \n",
       "5  Astronauts Dock With Space Station After Histo...  techcrunch       0.51   \n",
       "6  iOS 14 lets deaf users set alerts for importan...     nytimes       0.48   \n",
       "\n",
       "  groundtruth  \n",
       "0     nytimes  \n",
       "1     nytimes  \n",
       "2      github  \n",
       "3      github  \n",
       "4  techcrunch  \n",
       "5     nytimes  \n",
       "6  techcrunch  "
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prev_month"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2020 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-gpu.2-1.m50",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-1:m50"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
